diff --git a/README.md b/README.md index 0a3e9c704..8e439ae5b 100644 --- a/README.md +++ b/README.md @@ -5,10 +5,10 @@ RAPIDS Accelerator for Apache Spark accelerates Spark applications with no code You can download the latest version of RAPIDS Accelerator [here](https://nvidia.github.io/spark-rapids/docs/download.html). This repo contains examples and applications that showcases the performance and benefits of using RAPIDS Accelerator in data processing and machine learning pipelines. -There are broadly four categories of examples in this repo: +There are broadly five categories of examples in this repo: 1. [SQL/Dataframe](./examples/SQL+DF-Examples) 2. [Spark XGBoost](./examples/XGBoost-Examples) -3. [Deep Learning/Machine Learning](./examples/ML+DL-Examples) +3. [Machine Learning/Deep Learning](./examples/ML+DL-Examples) 4. [RAPIDS UDF](./examples/UDF-Examples) 5. [Databricks Tools demo notebooks](./tools/databricks) @@ -23,7 +23,8 @@ Here is the list of notebooks in this repo: | 3 | XGBoost | Agaricus (Scala) | Uses XGBoost classifier function to create model that can accurately differentiate between edible and poisonous mushrooms with the [agaricus dataset](https://archive.ics.uci.edu/ml/datasets/mushroom) | 4 | XGBoost | Mortgage (Scala) | End-to-end ETL + XGBoost example to predict mortgage default with [Fannie Mae Single-Family Loan Performance Data](https://capitalmarkets.fanniemae.com/credit-risk-transfer/single-family-credit-risk-transfer/fannie-mae-single-family-loan-performance-data) | 5 | XGBoost | Taxi (Scala) | End-to-end ETL + XGBoost example to predict taxi trip fare amount with [NYC taxi trips data set](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page) -| 6 | ML/DL | PCA End-to-End | Spark MLlib based PCA example to train and transform with a synthetic dataset +| 6 | ML/DL | PCA | [Spark-Rapids-ML](https://github.com/NVIDIA/spark-rapids-ml) based PCA example to train and transform with a synthetic dataset +| 7 | ML/DL | DL Inference | 11 notebooks demonstrating distributed model inference on Spark using the `predict_batch_udf` across various frameworks: PyTorch, HuggingFace, and TensorFlow Here is the list of Apache Spark applications (Scala and PySpark) that can be built for running on GPU with RAPIDS Accelerator in this repo: @@ -33,7 +34,7 @@ can be built for running on GPU with RAPIDS Accelerator in this repo: | 1 | XGBoost | Agaricus (Scala) | Uses XGBoost classifier function to create model that can accurately differentiate between edible and poisonous mushrooms with the [agaricus dataset](https://archive.ics.uci.edu/ml/datasets/mushroom) | 2 | XGBoost | Mortgage (Scala) | End-to-end ETL + XGBoost example to predict mortgage default with [Fannie Mae Single-Family Loan Performance Data](https://capitalmarkets.fanniemae.com/credit-risk-transfer/single-family-credit-risk-transfer/fannie-mae-single-family-loan-performance-data) | 3 | XGBoost | Taxi (Scala) | End-to-end ETL + XGBoost example to predict taxi trip fare amount with [NYC taxi trips data set](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page) -| 4 | ML/DL | PCA End-to-End | Spark MLlib based PCA example to train and transform with a synthetic dataset +| 4 | ML/DL | PCA | [Spark-Rapids-ML](https://github.com/NVIDIA/spark-rapids-ml) based PCA example to train and transform with a synthetic dataset | 5 | UDF | URL Decode | Decodes URL-encoded strings using the [Java APIs of RAPIDS cudf](https://docs.rapids.ai/api/cudf-java/legacy/) | 6 | UDF | URL Encode | URL-encodes strings using the [Java APIs of RAPIDS cudf](https://docs.rapids.ai/api/cudf-java/legacy/) | 7 | UDF | [CosineSimilarity](./examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/java/com/nvidia/spark/rapids/udf/java/CosineSimilarity.java) | Computes the cosine similarity between two float vectors using [native code](./examples/UDF-Examples/RAPIDS-accelerated-UDFs/src/main/cpp/src) diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/README.md b/examples/ML+DL-Examples/Spark-DL/dl_inference/README.md index 177af9072..782db87c6 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/README.md +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/README.md @@ -30,17 +30,49 @@ predictions = df.withColumn("preds", mnist("data")).collect() In this simple case, the `predict_batch_fn` will use TensorFlow APIs to load the model and return a simple `predict` function which operates on numpy arrays. The `predict_batch_udf` will automatically convert the Spark DataFrame columns to the expected numpy inputs. -All notebooks have been saved with sample outputs for quick browsing. +All notebooks have been saved with sample outputs for quick browsing. +Here is a full list of the notebooks with their published example links: + +| | Category | Notebook Name | Description | Link +| ------------- | ------------- | ------------- | ------------- | ------------- +| 1 | PyTorch | Image Classification | Training a model to predict clothing categories in FashionMNIST, including accelerated inference with Torch-TensorRT. | [Link](https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html) +| 2 | PyTorch | Regression | Training a model to predict housing prices in the California Housing Dataset, including accelerated inference with Torch-TensorRT. | [Link](https://github.com/christianversloot/machine-learning-articles/blob/main/how-to-create-a-neural-network-for-regression-with-pytorch.md) +| 3 | Tensorflow | Image Classification | Training a model to predict hand-written digits in MNIST. | [Link](https://www.tensorflow.org/tutorials/keras/save_and_load) +| 4 | Tensorflow | Feature Columns | Training a model with preprocessing layers to predict likelihood of pet adoption in the PetFinder mini dataset. | [Link](https://www.tensorflow.org/tutorials/structured_data/preprocessing_layers) +| 5 | Tensorflow | Keras Metadata | Training ResNet-50 to perform flower recognition on Databricks. | [Link](https://docs.databricks.com/en/_extras/notebooks/source/deep-learning/keras-metadata.html) +| 6 | Tensorflow | Text Classification | Training a model to perform sentiment analysis on the IMDB dataset. | [Link](https://www.tensorflow.org/tutorials/keras/text_classification) +| 7+8 | HuggingFace | Conditional Generation | Sentence translation using the T5 text-to-text transformer, with notebooks demoing both Torch and Tensorflow. | [Link](https://huggingface.co/docs/transformers/model_doc/t5#t5) +| 9+10 | HuggingFace | Pipelines | Sentiment analysis using Huggingface pipelines, with notebooks demoing both Torch and Tensorflow. | [Link](https://huggingface.co/docs/transformers/quicktour#pipeline-usage) +| 11 | HuggingFace | Sentence Transformers | Sentence embeddings using the SentenceTransformers framework in Torch. | [Link](https://huggingface.co/sentence-transformers) ## Running the Notebooks If you want to run the notebooks yourself, please follow these instructions. -**Note**: for demonstration purposes, these examples just use a local Spark Standalone cluster with a single executor, but you should be able to run them on any distributed Spark cluster. +**Notes**: +- The notebooks require a GPU environment for the executors. +- Please create separate environments for PyTorch and Tensorflow examples as specified below. This will avoid conflicts between the CUDA libraries bundled with their respective versions. The Huggingface examples will have a `_torch` or `_tf` suffix to specify the environment used. +- The PyTorch notebooks include model compilation and accelerated inference with TensorRT. While not included in the notebooks, Tensorflow also supports [integration with TensorRT](https://docs.nvidia.com/deeplearning/frameworks/tf-trt-user-guide/index.html), but may require downgrading the TF version. +- For demonstration purposes, these examples just use a local Spark Standalone cluster with a single executor, but you should be able to run them on any distributed Spark cluster. + +#### Create environment + +**For PyTorch:** +``` +conda create -n spark-dl-torch python=3.11 +conda activate spark-dl-torch +pip install -r torch_requirements.txt +``` +**For TensorFlow:** ``` -# install dependencies for example notebooks -pip install -r requirements.txt +conda create -n spark-dl-tf python=3.11 +conda activate spark-dl-tf +pip install -r tf_requirements.txt +``` + +#### Launch Jupyter + Spark +``` # setup environment variables export SPARK_HOME=/path/to/spark export MASTER=spark://$(hostname):7077 @@ -70,4 +102,4 @@ The example notebooks also demonstrate integration with [Triton Inference Server **Note**: Some examples may require special configuration of server as highlighted in the notebooks. -**Note**: for demonstration purposes, the Triton Inference Server integrations just launch the server in a docker container on the local host, so you will need to [install docker](https://docs.docker.com/engine/install/) on your local host. Most real-world deployments will likely be hosted on remote machines. \ No newline at end of file +**Note**: for demonstration purposes, the Triton Inference Server integrations just launch the server in a docker container on the local host, so you will need to [install docker](https://docs.docker.com/engine/install/) on your local host. Most real-world deployments will likely be hosted on remote machines. diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/conditional_generation_tf.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/conditional_generation_tf.ipynb new file mode 100644 index 000000000..e2c67eb98 --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/conditional_generation_tf.ipynb @@ -0,0 +1,1795 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "777fc40d", + "metadata": {}, + "source": [ + "# PySpark Huggingface Inferencing\n", + "## Conditional generation with Tensorflow\n", + "\n", + "From: https://huggingface.co/docs/transformers/model_doc/t5" + ] + }, + { + "cell_type": "markdown", + "id": "05c79ac4-bf25-421e-b55e-020d6d9e15d5", + "metadata": {}, + "source": [ + "### Using TensorFlow\n", + "Note that cuFFT/cuDNN/cuBLAS registration errors are expected with `tf=2.17.0` and will not affect behavior, as noted in [this issue.](https://github.com/tensorflow/tensorflow/issues/62075) \n", + "This notebook does not demonstrate inference with TensorRT, as [TF-TRT](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#tensorrt-10) does not yet support `tf=2.17.0`. See the `pytorch` notebooks for TensorRT demos." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "f6f0dbf3-712b-4c58-85eb-261ce15bb2be", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-11 00:16:59.451769: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", + "2024-10-11 00:16:59.459246: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", + "2024-10-11 00:16:59.467162: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", + "2024-10-11 00:16:59.469569: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n", + "2024-10-11 00:16:59.475888: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", + "To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", + "2024-10-11 00:16:59.818338: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" + ] + } + ], + "source": [ + "from transformers import AutoTokenizer, TFT5ForConditionalGeneration" + ] + }, + { + "cell_type": "markdown", + "id": "5346a20c", + "metadata": {}, + "source": [ + "Enabling Huggingface tokenizer parallelism so that it is not automatically disabled with Python parallelism. See [this thread](https://github.com/huggingface/transformers/issues/5486) for more info. " + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "a1008e27", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"true\"" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "275890d7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2.17.0\n" + ] + } + ], + "source": [ + "import tensorflow as tf\n", + "\n", + "# Enable GPU memory growth\n", + "gpus = tf.config.experimental.list_physical_devices('GPU')\n", + "if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + " \n", + "print(tf.__version__)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "2684fb41-9467-40c0-9d7e-a1cc867c5a3c", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-11 00:17:00.886565: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2021] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 46024 MB memory: -> device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:01:00.0, compute capability: 8.6\n", + "All PyTorch model weights were used when initializing TFT5ForConditionalGeneration.\n", + "\n", + "All the weights of TFT5ForConditionalGeneration were initialized from the PyTorch model.\n", + "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFT5ForConditionalGeneration for predictions without further training.\n" + ] + } + ], + "source": [ + "tokenizer = AutoTokenizer.from_pretrained(\"google-t5/t5-small\")\n", + "model = TFT5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n", + "\n", + "task_prefix = \"translate English to German: \"\n", + "\n", + "lines = [\n", + " \"The house is wonderful\",\n", + " \"Welcome to NYC\",\n", + " \"HuggingFace is a company\"\n", + "]\n", + "\n", + "input_sequences = [task_prefix + l for l in lines]" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "6eb2dfdb-0ad3-4d0f-81a4-268d92c53759", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", + "I0000 00:00:1728605822.106234 276792 service.cc:146] XLA service 0x7f53a8003630 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n", + "I0000 00:00:1728605822.106259 276792 service.cc:154] StreamExecutor device (0): NVIDIA RTX A6000, Compute Capability 8.6\n", + "2024-10-11 00:17:02.108842: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n", + "2024-10-11 00:17:02.117215: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 8907\n", + "I0000 00:00:1728605822.137920 276792 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n" + ] + } + ], + "source": [ + "input_ids = tokenizer(input_sequences, \n", + " padding=\"longest\", \n", + " max_length=512,\n", + " truncation=True,\n", + " return_tensors=\"tf\").input_ids\n", + "outputs = model.generate(input_ids, max_length=20)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "720158d4-e0e0-4904-b096-e5aede756afd", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['Das Haus ist wunderbar',\n", + " 'Willkommen in NYC',\n", + " 'HuggingFace ist ein Unternehmen']" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "[tokenizer.decode(o, skip_special_tokens=True) for o in outputs]" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "8d4b364b-13cb-48ea-a97a-ccfc9e408075", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'tf'" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "model.framework" + ] + }, + { + "cell_type": "markdown", + "id": "546eabe0", + "metadata": {}, + "source": [ + "## PySpark" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "2f6db1f0-7d68-4af7-8bd6-c9fa45906c61", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from pathlib import Path\n", + "from datasets import load_dataset" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "68121304-f1df-466e-9347-c9d2b36a9b3a", + "metadata": {}, + "outputs": [], + "source": [ + "from pyspark.sql.types import *\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf\n", + "import socket" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "6279a849", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "24/10/11 00:17:03 WARN Utils: Your hostname, cb4ae00-lcedt resolves to a loopback address: 127.0.1.1; using 10.110.47.100 instead (on interface eno1)\n", + "24/10/11 00:17:03 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n", + "Setting default log level to \"WARN\".\n", + "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n", + "24/10/11 00:17:03 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n" + ] + } + ], + "source": [ + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "hostname = socket.gethostname()\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.maxRecordsPerBatch\", \"512\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "b8453111-d068-49bb-ab91-8ae3d8bcdb7a", + "metadata": {}, + "outputs": [], + "source": [ + "# load IMDB reviews (test) dataset\n", + "data = load_dataset(\"imdb\", split=\"test\")" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "7ad01d4a", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "25000" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "lines = []\n", + "for example in data:\n", + " lines.append([example[\"text\"].split(\".\")[0]])\n", + "\n", + "len(lines)" + ] + }, + { + "cell_type": "markdown", + "id": "6fd5b472-47e8-4804-9907-772793fedb2b", + "metadata": {}, + "source": [ + "### Create PySpark DataFrame" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "d24d9404-0269-476e-a9dd-1842667c915a", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "StructType([StructField('lines', StringType(), True)])" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = spark.createDataFrame(lines, ['lines']).repartition(8)\n", + "df.schema" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "4384c762-1f79-4f60-876c-94b1f552e8fb", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "[Row(lines='(Some Spoilers) Dull as dishwater slasher flick that has this deranged homeless man Harry, Darwyn Swalve, out murdering real-estate agent all over the city of L')]" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df.take(1)" + ] + }, + { + "cell_type": "markdown", + "id": "42ba3513-82dd-47e7-8193-eb4389458757", + "metadata": {}, + "source": [ + "### Save the test dataset as parquet files" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "e7eec8ec-4126-4890-b957-025809fad67d", + "metadata": {}, + "outputs": [], + "source": [ + "df.write.mode(\"overwrite\").parquet(\"imdb_test\")" + ] + }, + { + "cell_type": "markdown", + "id": "304e1fc8-42a3-47dd-b3c0-47efd5be1040", + "metadata": {}, + "source": [ + "### Check arrow memory configuration" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "20554ea5-01be-4a30-8607-db5d87786fec", + "metadata": {}, + "outputs": [], + "source": [ + "if int(spark.conf.get(\"spark.sql.execution.arrow.maxRecordsPerBatch\")) > 512:\n", + " print(\"Decreasing `spark.sql.execution.arrow.maxRecordsPerBatch` to ensure the vectorized reader won't run out of memory\")\n", + " spark.conf.set(\"spark.sql.execution.arrow.maxRecordsPerBatch\", \"512\")\n", + "assert len(df.head()) > 0, \"`df` should not be empty\"" + ] + }, + { + "cell_type": "markdown", + "id": "06a4ecab-c9d9-466f-ba49-902ad1fd5488", + "metadata": {}, + "source": [ + "## Inference using Spark DL API\n", + "Note: you can restart the kernel and run from this point to simulate running in a different node or environment." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "e7a00479-1347-4de8-8431-faa77f8cdf4c", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import col, pandas_udf, struct\n", + "from pyspark.sql.types import StringType" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "b9a0889a-35b4-493a-8197-1146fc7efd53", + "metadata": {}, + "outputs": [], + "source": [ + "# only use first sentence and add prefix for conditional generation\n", + "def preprocess(text: pd.Series, prefix: str = \"\") -> pd.Series:\n", + " @pandas_udf(\"string\")\n", + " def _preprocess(text: pd.Series) -> pd.Series:\n", + " return pd.Series([prefix + s.split(\".\")[0] for s in text])\n", + " return _preprocess(text)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "c483e4d4-9ab1-416f-a766-694e17490fd3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| lines|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| I am a big fan of The ABC Movies of the Week genre|\n", + "|In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Full House\" and the long-defunc...|\n", + "|When The Spirits Within was released, all you heard from Final Fantasy fans was how awful the movie was because it di...|\n", + "| I like to think of myself as a bad movie connoisseur|\n", + "|This film did well at the box office, and the producers of this mess thought the stars had such good chemistry in thi...|\n", + "|Following the pleasingly atmospheric original and the amusingly silly second one, this incredibly dull, slow, and une...|\n", + "| I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| I have read all of the reviews for this direct to video movie|\n", + "|Yes, it was an awful movie, but there was a song near the beginning of the movie, I think, called \"I got a Woody\" or ...|\n", + "| This was the most uninteresting horror flick I have seen to date|\n", + "|I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'The French Connection\", but it...|\n", + "|Heart of Darkness Movie Review Could a book that is well known for its eloquent wording and complicated concepts ever...|\n", + "| A bad movie ABOUT a bad movie|\n", + "|Apart from the fact that this film was made ( I suppose it seemed a good idea at the time considering BOTTOM was so p...|\n", + "|Watching this movie, you just have to ask: What were they thinking? There are so many noticeably bad parts of this mo...|\n", + "| OK, lets start with the best|\n", + "| Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 years|\n", + "| C|\n", + "| Tom and Jerry are transporting goods via airplane to Africa|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + }, + { + "data": { + "text/plain": [ + "100" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# only use first N examples, since this is slow\n", + "df = spark.read.parquet(\"imdb_test\").limit(100)\n", + "df.show(truncate=120)\n", + "df.count()" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "831bc52c-a5c6-4c29-a6da-0566b5167773", + "metadata": {}, + "outputs": [], + "source": [ + "# only use first 100 rows, since generation takes a while\n", + "df1 = df.withColumn(\"input\", preprocess(col(\"lines\"), \"Translate English to German: \")).select(\"input\").limit(100).cache()" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "46dac59c-5a54-4576-91e0-279c8b375b95", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "100" + ] + }, + "execution_count": 21, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df1.count()" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "fef1d846-5852-4762-8527-602f32c0d7cd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| input|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| Translate English to German: This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| Translate English to German: I am a big fan of The ABC Movies of the Week genre|\n", + "|Translate English to German: In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Fu...|\n", + "|Translate English to German: When The Spirits Within was released, all you heard from Final Fantasy fans was how awfu...|\n", + "| Translate English to German: I like to think of myself as a bad movie connoisseur|\n", + "|Translate English to German: This film did well at the box office, and the producers of this mess thought the stars h...|\n", + "|Translate English to German: Following the pleasingly atmospheric original and the amusingly silly second one, this i...|\n", + "| Translate English to German: I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| Translate English to German: I have read all of the reviews for this direct to video movie|\n", + "|Translate English to German: Yes, it was an awful movie, but there was a song near the beginning of the movie, I thin...|\n", + "| Translate English to German: This was the most uninteresting horror flick I have seen to date|\n", + "|Translate English to German: I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'T...|\n", + "|Translate English to German: Heart of Darkness Movie Review Could a book that is well known for its eloquent wording ...|\n", + "| Translate English to German: A bad movie ABOUT a bad movie|\n", + "|Translate English to German: Apart from the fact that this film was made ( I suppose it seemed a good idea at the tim...|\n", + "|Translate English to German: Watching this movie, you just have to ask: What were they thinking? There are so many no...|\n", + "| Translate English to German: OK, lets start with the best|\n", + "|Translate English to German: Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 y...|\n", + "| Translate English to German: C|\n", + "| Translate English to German: Tom and Jerry are transporting goods via airplane to Africa|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + } + ], + "source": [ + "df1.show(truncate=120)" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "e7ae69d3-70c2-4765-928f-c96a7ba59829", + "metadata": {}, + "outputs": [], + "source": [ + "def predict_batch_fn():\n", + " import tensorflow as tf\n", + " import numpy as np\n", + " from transformers import TFT5ForConditionalGeneration, AutoTokenizer\n", + "\n", + " # Enable GPU memory growth\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + "\n", + " model = TFT5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n", + " tokenizer = AutoTokenizer.from_pretrained(\"google-t5/t5-small\")\n", + "\n", + " def predict(inputs):\n", + " flattened = np.squeeze(inputs).tolist() # convert 2d numpy array of string into flattened python list\n", + " input_ids = tokenizer(flattened, \n", + " padding=\"longest\", \n", + " max_length=512,\n", + " return_tensors=\"tf\").input_ids\n", + " output_ids = model.generate(input_ids, max_length=20)\n", + " string_outputs = np.array([tokenizer.decode(o, skip_special_tokens=True) for o in output_ids])\n", + " print(\"predict: {}\".format(len(flattened)))\n", + "\n", + " return string_outputs\n", + " \n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "36684f59-d947-43f8-a2e8-c7a423764e88", + "metadata": {}, + "outputs": [], + "source": [ + "generate = predict_batch_udf(predict_batch_fn,\n", + " return_type=StringType(),\n", + " batch_size=10)" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "6a01c855-8fa1-4765-a3a5-2c9dd872df10", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 21:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 9.39 ms, sys: 2.14 ms, total: 11.5 ms\n", + "Wall time: 11.4 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# first pass caches model/fn\n", + "preds = df1.withColumn(\"preds\", generate(struct(\"input\")))\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "d912d4b0-cd0b-44ea-859a-b23455cc2700", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 23:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 3.62 ms, sys: 4.01 ms, total: 7.64 ms\n", + "Wall time: 8.53 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df1.withColumn(\"preds\", generate(\"input\"))\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "5fe3d88b-30f7-468f-8db8-1f4118d0f26c", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 25:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 5.37 ms, sys: 2.51 ms, total: 7.88 ms\n", + "Wall time: 8.52 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df1.withColumn(\"preds\", generate(col(\"input\")))\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "4ad9b365-4b9a-438e-8fdf-47da55cb1cf4", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 27:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "| input| preds|\n", + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "|Translate English to German: This is so overly clichéd yo...| Das ist so übertrieben klischeehaft, dass Sie es nach den|\n", + "|Translate English to German: I am a big fan of The ABC Mo...| Ich bin ein großer Fan von The ABC Movies of the Week|\n", + "|Translate English to German: In the early 1990's \"Step-by...| Anfang der 1990er Jahre kam \"Step-by-Step\" als müh|\n", + "|Translate English to German: When The Spirits Within was ...|Als The Spirits Within veröffentlicht wurde, hörten Sie v...|\n", + "|Translate English to German: I like to think of myself as...| Ich halte mich gerne als schlechter Filmliebhaber|\n", + "|Translate English to German: This film did well at the bo...|Dieser Film hat sich gut an der Boxoffice ereignet, und d...|\n", + "|Translate English to German: Following the pleasingly atm...|Nach dem erfreulich stimmungsvollen Original und dem amüsant|\n", + "|Translate English to German: I like CKY and Viva La Bam, ...| Ich mag CKY und Viva La Bam, also konnte ich mich nicht|\n", + "|Translate English to German: I have read all of the revie...| Ich habe alle Rezensionen zu diesem direkten Film gelesen.|\n", + "|Translate English to German: Yes, it was an awful movie, ...| Ja, es war ein schrecklicher Film, aber es gab|\n", + "|Translate English to German: This was the most uninterest...|Dies war der größte Horrorfilm, den ich bisher gesehen habe.|\n", + "|Translate English to German: I don't know if this excepti...|Ich weiß nicht, ob dieser außergewöhnlich langweilige Fil...|\n", + "|Translate English to German: Heart of Darkness Movie Revi...|Herz der Dunkelheit Film Review Kann ein Buch, das für se...|\n", + "| Translate English to German: A bad movie ABOUT a bad movie| Ein schlechter Film ABOUT a bad movie|\n", + "|Translate English to German: Apart from the fact that thi...| Dieser Film wurde zwar fertiggestellt, aber es schien mir |\n", + "|Translate English to German: Watching this movie, you jus...|Wenn man diesen Film anschaut, muss man einfach fragen: W...|\n", + "| Translate English to German: OK, lets start with the best| OK, lets start with the best|\n", + "|Translate English to German: Anna Christie (Greta Garbo) ...| Anna Christie (Greta Garbo) kehrt nach 15 Jahren zurück,|\n", + "| Translate English to German: C| C|\n", + "|Translate English to German: Tom and Jerry are transporti...|Tom und Jerry transportieren Güter über Flugzeug nach Afrika|\n", + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "preds.show(truncate=60)" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "1eb0c83b-d91b-4f8c-a5e7-c35f55c88108", + "metadata": {}, + "outputs": [], + "source": [ + "# only use first 100 rows, since generation takes a while\n", + "df2 = df.withColumn(\"input\", preprocess(col(\"lines\"), \"Translate English to French: \")).select(\"input\").limit(100).cache()" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "054f94fd-fe79-41e7-b1c7-6124083acc72", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| input|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| Translate English to French: This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| Translate English to French: I am a big fan of The ABC Movies of the Week genre|\n", + "|Translate English to French: In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Fu...|\n", + "|Translate English to French: When The Spirits Within was released, all you heard from Final Fantasy fans was how awfu...|\n", + "| Translate English to French: I like to think of myself as a bad movie connoisseur|\n", + "|Translate English to French: This film did well at the box office, and the producers of this mess thought the stars h...|\n", + "|Translate English to French: Following the pleasingly atmospheric original and the amusingly silly second one, this i...|\n", + "| Translate English to French: I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| Translate English to French: I have read all of the reviews for this direct to video movie|\n", + "|Translate English to French: Yes, it was an awful movie, but there was a song near the beginning of the movie, I thin...|\n", + "| Translate English to French: This was the most uninteresting horror flick I have seen to date|\n", + "|Translate English to French: I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'T...|\n", + "|Translate English to French: Heart of Darkness Movie Review Could a book that is well known for its eloquent wording ...|\n", + "| Translate English to French: A bad movie ABOUT a bad movie|\n", + "|Translate English to French: Apart from the fact that this film was made ( I suppose it seemed a good idea at the tim...|\n", + "|Translate English to French: Watching this movie, you just have to ask: What were they thinking? There are so many no...|\n", + "| Translate English to French: OK, lets start with the best|\n", + "|Translate English to French: Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 y...|\n", + "| Translate English to French: C|\n", + "| Translate English to French: Tom and Jerry are transporting goods via airplane to Africa|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + } + ], + "source": [ + "df2.show(truncate=120)" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "6f6b70f9-188a-402b-9143-78a5788140e4", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 33:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 2.9 ms, sys: 5.97 ms, total: 8.87 ms\n", + "Wall time: 11.7 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# first pass caches model/fn\n", + "preds = df2.withColumn(\"preds\", generate(struct(\"input\")))\n", + "result = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "031a6a5e-7999-4653-b394-19ed478d8c96", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 35:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 4.41 ms, sys: 1.59 ms, total: 5.99 ms\n", + "Wall time: 8.23 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df2.withColumn(\"preds\", generate(\"input\"))\n", + "result = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "229b6515-82f6-4e9c-90f0-a9c3cfb26301", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 37:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 5.46 ms, sys: 1.17 ms, total: 6.63 ms\n", + "Wall time: 8.08 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df2.withColumn(\"preds\", generate(col(\"input\")))\n", + "result = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "8be750ac-fa39-452e-bb4c-c2270bc2f70d", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 39:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "| input| preds|\n", + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "|Translate English to French: This is so overly clichéd yo...| Vous ne pouvez pas en tirer d'un tel cliché|\n", + "|Translate English to French: I am a big fan of The ABC Mo...| Je suis un grand fan du genre The ABC Movies of the Week|\n", + "|Translate English to French: In the early 1990's \"Step-by...| Au début des années 1990, «Step-by-Step» a été une|\n", + "|Translate English to French: When The Spirits Within was ...|Lorsque The Spirits Within a été publié, tout ce que vous...|\n", + "|Translate English to French: I like to think of myself as...| Je me considère comme un mauvais réalisateur de films|\n", + "|Translate English to French: This film did well at the bo...| Ce film a bien avancé à la salle de cinéma et|\n", + "|Translate English to French: Following the pleasingly atm...|Après l'original agréablement atmosphérique et la seconde...|\n", + "|Translate English to French: I like CKY and Viva La Bam, ...| Je m'aime CKY et Viva La Bam, |\n", + "|Translate English to French: I have read all of the revie...| J'ai lu tous les commentaires pour ce film direct à vidéo|\n", + "|Translate English to French: Yes, it was an awful movie, ...| Oui, c'était un film terrible, mais il y avait une chanson|\n", + "|Translate English to French: This was the most uninterest...| Ce fut le plus inquiétant et le plus inquiétant d'h|\n", + "|Translate English to French: I don't know if this excepti...|Je ne sais pas si ce film extrêmement tacheté était desti...|\n", + "|Translate English to French: Heart of Darkness Movie Revi...| Un livre connu pour son éloquence et ses concepts complexes|\n", + "| Translate English to French: A bad movie ABOUT a bad movie| Un mauvais film ABOUT a bad movie|\n", + "|Translate English to French: Apart from the fact that thi...|En plus du fait que ce film a été réalisé (je suppose quil s|\n", + "|Translate English to French: Watching this movie, you jus...|Vous devez simplement vous demander : « Que pense-t-il? » Il|\n", + "| Translate English to French: OK, lets start with the best| OK, s'il y a lieu de commencer par le meilleur|\n", + "|Translate English to French: Anna Christie (Greta Garbo) ...|Anna Christie (Greta Garbo) retourne pour voir son père C...|\n", + "| Translate English to French: C| C|\n", + "|Translate English to French: Tom and Jerry are transporti...| Tom et Jerry transportent des marchandises par avion en |\n", + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "preds.show(truncate=60)" + ] + }, + { + "cell_type": "markdown", + "id": "bcabb2a8-3880-46ec-8e01-5a10f71fe83d", + "metadata": {}, + "source": [ + "### Using Triton Inference Server\n", + "\n", + "Note: you can restart the kernel and run from this point to simulate running in a different node or environment. " + ] + }, + { + "cell_type": "markdown", + "id": "5d98fa52-7665-49bf-865a-feec86effe23", + "metadata": {}, + "source": [ + "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments) with the compatible versions of Python/Numpy for Triton 24.08, using a conda-pack environment created as follows:\n", + "```\n", + "conda create -n huggingface-tf -c conda-forge python=3.10.0\n", + "conda activate huggingface-tf\n", + "\n", + "export PYTHONNOUSERSITE=True\n", + "pip install numpy==1.26.4 tensorflow[and-cuda] tf-keras transformers conda-pack \n", + "\n", + "conda-pack # huggingface-tf.tar.gz\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "b858cf85-82e6-41ef-905b-d8c5d6fea492", + "metadata": {}, + "outputs": [], + "source": [ + "import os" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "05ce7c77-d562-45e8-89bb-cd656aba5a5f", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "# copy custom model to expected layout for Triton\n", + "rm -rf models\n", + "mkdir -p models\n", + "cp -r models_config/hf_generation_tf models\n", + "\n", + "# add custom execution environment\n", + "cp huggingface-tf.tar.gz models" + ] + }, + { + "cell_type": "markdown", + "id": "a552865c-5dad-4f25-8834-f41e253ac2f6", + "metadata": { + "tags": [] + }, + "source": [ + "#### Start Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "afd00b7e-8150-4c95-a2e4-037e9c90f92a", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "[True]" + ] + }, + "execution_count": 37, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "num_executors = 1\n", + "triton_models_dir = \"{}/models\".format(os.getcwd())\n", + "huggingface_cache_dir = \"{}/.cache/huggingface\".format(os.path.expanduser('~'))\n", + "nodeRDD = sc.parallelize(list(range(num_executors)), num_executors)\n", + "\n", + "def start_triton(it):\n", + " import docker\n", + " import time\n", + " import tritonclient.grpc as grpcclient\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " if containers:\n", + " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", + " else:\n", + " try:\n", + " container=client.containers.run(\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", + " detach=True,\n", + " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", + " environment=[\n", + " \"TRANSFORMERS_CACHE=/cache\"\n", + " ],\n", + " name=\"spark-triton\",\n", + " network_mode=\"host\",\n", + " remove=True,\n", + " shm_size=\"1G\",\n", + " volumes={\n", + " triton_models_dir: {\"bind\": \"/models\", \"mode\": \"ro\"},\n", + " huggingface_cache_dir: {\"bind\": \"/cache\", \"mode\": \"rw\"}\n", + " }\n", + " )\n", + " print(\">>>> starting triton: {}\".format(container.short_id))\n", + " except Exception as e:\n", + " print(\">>>> failed to start triton: {}\".format(e))\n", + " # wait for triton to be running\n", + " time.sleep(15)\n", + " client = grpcclient.InferenceServerClient(\"localhost:8001\")\n", + " ready = False\n", + " while not ready:\n", + " try:\n", + " ready = client.is_server_ready()\n", + " except Exception as e:\n", + " time.sleep(5)\n", + "\n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(start_triton).collect()" + ] + }, + { + "cell_type": "markdown", + "id": "528d2df6-49fc-4be7-a534-a087dfe31c84", + "metadata": {}, + "source": [ + "#### Run inference" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "1a997c33-5202-466d-8304-b8c30f32978f", + "metadata": { + "tags": [] + }, + "outputs": [], + "source": [ + "import pandas as pd\n", + "from functools import partial\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import col, pandas_udf, struct\n", + "from pyspark.sql.types import StringType" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "9dea1875-6b95-4fc0-926d-a625a441b33d", + "metadata": {}, + "outputs": [], + "source": [ + "# only use first N examples, since this is slow\n", + "df = spark.read.parquet(\"imdb_test\").limit(100).cache()" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "5d6c54e7-534d-406f-b8e6-fd592efd0ab2", + "metadata": {}, + "outputs": [], + "source": [ + "# only use first sentence and add prefix for conditional generation\n", + "def preprocess(text: pd.Series, prefix: str = \"\") -> pd.Series:\n", + " @pandas_udf(\"string\")\n", + " def _preprocess(text: pd.Series) -> pd.Series:\n", + " return pd.Series([prefix + s.split(\".\")[0] for s in text])\n", + " return _preprocess(text)" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "dc1bbbe3-4232-49e5-80f6-99976524b73b", + "metadata": {}, + "outputs": [], + "source": [ + "# only use first 100 rows, since generation takes a while\n", + "df1 = df.withColumn(\"input\", preprocess(col(\"lines\"), \"Translate English to German: \")).select(\"input\").limit(100)" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "id": "5d10c61c-6102-4d19-8dd6-0c7b5b65343e", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| input|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| Translate English to German: This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| Translate English to German: I am a big fan of The ABC Movies of the Week genre|\n", + "|Translate English to German: In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Fu...|\n", + "|Translate English to German: When The Spirits Within was released, all you heard from Final Fantasy fans was how awfu...|\n", + "| Translate English to German: I like to think of myself as a bad movie connoisseur|\n", + "|Translate English to German: This film did well at the box office, and the producers of this mess thought the stars h...|\n", + "|Translate English to German: Following the pleasingly atmospheric original and the amusingly silly second one, this i...|\n", + "| Translate English to German: I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| Translate English to German: I have read all of the reviews for this direct to video movie|\n", + "|Translate English to German: Yes, it was an awful movie, but there was a song near the beginning of the movie, I thin...|\n", + "| Translate English to German: This was the most uninteresting horror flick I have seen to date|\n", + "|Translate English to German: I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'T...|\n", + "|Translate English to German: Heart of Darkness Movie Review Could a book that is well known for its eloquent wording ...|\n", + "| Translate English to German: A bad movie ABOUT a bad movie|\n", + "|Translate English to German: Apart from the fact that this film was made ( I suppose it seemed a good idea at the tim...|\n", + "|Translate English to German: Watching this movie, you just have to ask: What were they thinking? There are so many no...|\n", + "| Translate English to German: OK, lets start with the best|\n", + "|Translate English to German: Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 y...|\n", + "| Translate English to German: C|\n", + "| Translate English to German: Tom and Jerry are transporting goods via airplane to Africa|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + } + ], + "source": [ + "df1.show(truncate=120)" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "id": "2e0907da-a5d9-4c3b-9db4-ce5e70ca9bb4", + "metadata": {}, + "outputs": [], + "source": [ + "def triton_fn(triton_uri, model_name):\n", + " import numpy as np\n", + " import tritonclient.grpc as grpcclient\n", + " \n", + " np_types = {\n", + " \"BOOL\": np.dtype(np.bool_),\n", + " \"INT8\": np.dtype(np.int8),\n", + " \"INT16\": np.dtype(np.int16),\n", + " \"INT32\": np.dtype(np.int32),\n", + " \"INT64\": np.dtype(np.int64),\n", + " \"FP16\": np.dtype(np.float16),\n", + " \"FP32\": np.dtype(np.float32),\n", + " \"FP64\": np.dtype(np.float64),\n", + " \"FP64\": np.dtype(np.double),\n", + " \"BYTES\": np.dtype(object)\n", + " }\n", + "\n", + " client = grpcclient.InferenceServerClient(triton_uri)\n", + " model_meta = client.get_model_metadata(model_name)\n", + " \n", + " def predict(inputs):\n", + " if isinstance(inputs, np.ndarray):\n", + " # single ndarray input\n", + " request = [grpcclient.InferInput(model_meta.inputs[0].name, inputs.shape, model_meta.inputs[0].datatype)]\n", + " request[0].set_data_from_numpy(inputs.astype(np_types[model_meta.inputs[0].datatype]))\n", + " else:\n", + " # dict of multiple ndarray inputs\n", + " request = [grpcclient.InferInput(i.name, inputs[i.name].shape, i.datatype) for i in model_meta.inputs]\n", + " for i in request:\n", + " i.set_data_from_numpy(inputs[i.name()].astype(np_types[i.datatype()]))\n", + " \n", + " response = client.infer(model_name, inputs=request)\n", + " \n", + " if len(model_meta.outputs) > 1:\n", + " # return dictionary of numpy arrays\n", + " return {o.name: response.as_numpy(o.name) for o in model_meta.outputs}\n", + " else:\n", + " # return single numpy array\n", + " return response.as_numpy(model_meta.outputs[0].name)\n", + " \n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 44, + "id": "9308bdd7-6f67-484d-8b51-dd1e1b2960ba", + "metadata": {}, + "outputs": [], + "source": [ + "generate = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"hf_generation_tf\"),\n", + " return_type=StringType(),\n", + " input_tensor_shapes=[[1]],\n", + " batch_size=100)" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "id": "38484ffd-370d-492b-8ca4-9eff9f242a9f", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 45:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 5.88 ms, sys: 3.96 ms, total: 9.84 ms\n", + "Wall time: 2.66 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# first pass caches model/fn\n", + "preds = df1.withColumn(\"preds\", generate(struct(\"input\")))\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "id": "ebcb6699-3ac2-4529-ab0f-fab0a5e792da", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 47:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 2.82 ms, sys: 1.05 ms, total: 3.87 ms\n", + "Wall time: 1.03 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df1.withColumn(\"preds\", generate(\"input\"))\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "id": "e2ed18ad-d00b-472c-b2c3-047932f2105d", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 49:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 1.55 ms, sys: 2.49 ms, total: 4.03 ms\n", + "Wall time: 967 ms\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df1.withColumn(\"preds\", generate(col(\"input\")))\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 48, + "id": "0cd64a1c-beb8-47d5-ac6f-e8525bb61176", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 51:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "| input| preds|\n", + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "|Translate English to German: This is so overly clichéd yo...| Das ist so übertrieben klischeehaft, dass Sie es nach den|\n", + "|Translate English to German: I am a big fan of The ABC Mo...| Ich bin ein großer Fan von The ABC Movies of the Week|\n", + "|Translate English to German: In the early 1990's \"Step-by...| Anfang der 1990er Jahre kam \"Step-by-Step\" als müh|\n", + "|Translate English to German: When The Spirits Within was ...|Als The Spirits Within veröffentlicht wurde, hörten Sie v...|\n", + "|Translate English to German: I like to think of myself as...| Ich halte mich gerne als schlechter Filmliebhaber|\n", + "|Translate English to German: This film did well at the bo...|Dieser Film hat sich gut an der Boxoffice ereignet, und d...|\n", + "|Translate English to German: Following the pleasingly atm...|Nach dem erfreulich stimmungsvollen Original und dem amüsant|\n", + "|Translate English to German: I like CKY and Viva La Bam, ...| Ich mag CKY und Viva La Bam, also konnte ich mich nicht|\n", + "|Translate English to German: I have read all of the revie...| Ich habe alle Rezensionen zu diesem direkten Film gelesen.|\n", + "|Translate English to German: Yes, it was an awful movie, ...| Ja, es war ein schrecklicher Film, aber es gab|\n", + "|Translate English to German: This was the most uninterest...|Dies war der größte Horrorfilm, den ich bisher gesehen habe.|\n", + "|Translate English to German: I don't know if this excepti...|Ich weiß nicht, ob dieser außergewöhnlich langweilige Fil...|\n", + "|Translate English to German: Heart of Darkness Movie Revi...|Herz der Dunkelheit Film Review Kann ein Buch, das für se...|\n", + "| Translate English to German: A bad movie ABOUT a bad movie| Ein schlechter Film ABOUT a bad movie|\n", + "|Translate English to German: Apart from the fact that thi...| Dieser Film wurde zwar fertiggestellt, aber es schien mir |\n", + "|Translate English to German: Watching this movie, you jus...|Wenn man diesen Film anschaut, muss man einfach fragen: W...|\n", + "| Translate English to German: OK, lets start with the best| OK, lets start with the best|\n", + "|Translate English to German: Anna Christie (Greta Garbo) ...| Anna Christie (Greta Garbo) kehrt nach 15 Jahren zurück,|\n", + "| Translate English to German: C| C|\n", + "|Translate English to German: Tom and Jerry are transporti...|Tom und Jerry transportieren Güter über Flugzeug nach Afrika|\n", + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "preds.show(truncate=60)" + ] + }, + { + "cell_type": "code", + "execution_count": 49, + "id": "af70fed8-0f2b-4ea7-841c-476afdf9b1c0", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "24/10/11 00:18:52 WARN CacheManager: Asked to cache already cached data.\n" + ] + } + ], + "source": [ + "# only use first 100 rows, since generation takes a while\n", + "df2 = df.withColumn(\"input\", preprocess(col(\"lines\"), \"Translate English to French: \")).select(\"input\").limit(100).cache()" + ] + }, + { + "cell_type": "code", + "execution_count": 50, + "id": "ef075e10-e22c-4236-9e0b-cb47cf2d3d06", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| input|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "| Translate English to French: This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| Translate English to French: I am a big fan of The ABC Movies of the Week genre|\n", + "|Translate English to French: In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Fu...|\n", + "|Translate English to French: When The Spirits Within was released, all you heard from Final Fantasy fans was how awfu...|\n", + "| Translate English to French: I like to think of myself as a bad movie connoisseur|\n", + "|Translate English to French: This film did well at the box office, and the producers of this mess thought the stars h...|\n", + "|Translate English to French: Following the pleasingly atmospheric original and the amusingly silly second one, this i...|\n", + "| Translate English to French: I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| Translate English to French: I have read all of the reviews for this direct to video movie|\n", + "|Translate English to French: Yes, it was an awful movie, but there was a song near the beginning of the movie, I thin...|\n", + "| Translate English to French: This was the most uninteresting horror flick I have seen to date|\n", + "|Translate English to French: I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'T...|\n", + "|Translate English to French: Heart of Darkness Movie Review Could a book that is well known for its eloquent wording ...|\n", + "| Translate English to French: A bad movie ABOUT a bad movie|\n", + "|Translate English to French: Apart from the fact that this film was made ( I suppose it seemed a good idea at the tim...|\n", + "|Translate English to French: Watching this movie, you just have to ask: What were they thinking? There are so many no...|\n", + "| Translate English to French: OK, lets start with the best|\n", + "|Translate English to French: Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 y...|\n", + "| Translate English to French: C|\n", + "| Translate English to French: Tom and Jerry are transporting goods via airplane to Africa|\n", + "+------------------------------------------------------------------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + } + ], + "source": [ + "df2.show(truncate=120)" + ] + }, + { + "cell_type": "code", + "execution_count": 51, + "id": "2e7e4af8-b815-4375-b851-8368309ee8e1", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 55:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 3.91 ms, sys: 1.34 ms, total: 5.25 ms\n", + "Wall time: 1.27 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df2.withColumn(\"preds\", generate(struct(\"input\")))\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 52, + "id": "7b0aefb0-a96b-4791-a23c-1ce9b24eb20c", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 57:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 4.31 ms, sys: 0 ns, total: 4.31 ms\n", + "Wall time: 1 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df2.withColumn(\"preds\", generate(\"input\"))\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 53, + "id": "1214b75b-a373-4579-b4c6-0cb8627da776", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 59:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 2.84 ms, sys: 1.31 ms, total: 4.15 ms\n", + "Wall time: 990 ms\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df2.withColumn(\"preds\", generate(col(\"input\")))\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 54, + "id": "c9dbd21f-9e37-4221-b765-80ba8c80b884", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 61:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "| input| preds|\n", + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "|Translate English to French: This is so overly clichéd yo...| Vous ne pouvez pas en tirer d'un tel cliché|\n", + "|Translate English to French: I am a big fan of The ABC Mo...| Je suis un grand fan du genre The ABC Movies of the Week|\n", + "|Translate English to French: In the early 1990's \"Step-by...| Au début des années 1990, «Step-by-Step» a été une|\n", + "|Translate English to French: When The Spirits Within was ...|Lorsque The Spirits Within a été publié, tout ce que vous...|\n", + "|Translate English to French: I like to think of myself as...| Je me considère comme un mauvais réalisateur de films|\n", + "|Translate English to French: This film did well at the bo...| Ce film a bien avancé à la salle de cinéma et|\n", + "|Translate English to French: Following the pleasingly atm...|Après l'original agréablement atmosphérique et la seconde...|\n", + "|Translate English to French: I like CKY and Viva La Bam, ...| Je m'aime CKY et Viva La Bam, |\n", + "|Translate English to French: I have read all of the revie...| J'ai lu tous les commentaires pour ce film direct à vidéo|\n", + "|Translate English to French: Yes, it was an awful movie, ...| Oui, c'était un film terrible, mais il y avait une chanson|\n", + "|Translate English to French: This was the most uninterest...| Ce fut le plus inquiétant et le plus inquiétant d'h|\n", + "|Translate English to French: I don't know if this excepti...|Je ne sais pas si ce film extrêmement tacheté était desti...|\n", + "|Translate English to French: Heart of Darkness Movie Revi...| Un livre connu pour son éloquence et ses concepts complexes|\n", + "| Translate English to French: A bad movie ABOUT a bad movie| Un mauvais film ABOUT a bad movie|\n", + "|Translate English to French: Apart from the fact that thi...|En plus du fait que ce film a été réalisé (je suppose quil s|\n", + "|Translate English to French: Watching this movie, you jus...|Vous devez simplement vous demander : « Que pense-t-il? » Il|\n", + "| Translate English to French: OK, lets start with the best| OK, s'il y a lieu de commencer par le meilleur|\n", + "|Translate English to French: Anna Christie (Greta Garbo) ...|Anna Christie (Greta Garbo) retourne pour voir son père C...|\n", + "| Translate English to French: C| C|\n", + "|Translate English to French: Tom and Jerry are transporti...| Tom et Jerry transportent des marchandises par avion en |\n", + "+------------------------------------------------------------+------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "preds.show(truncate=60)" + ] + }, + { + "cell_type": "markdown", + "id": "919e3113-64dd-482a-9233-6607b3f63c1e", + "metadata": { + "tags": [] + }, + "source": [ + "#### Stop Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": 55, + "id": "425d3b28-7705-45ba-8a18-ad34fc895219", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "[True]" + ] + }, + "execution_count": 55, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def stop_triton(it):\n", + " import docker\n", + " import time\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " print(\">>>> stopping containers: {}\".format([c.short_id for c in containers]))\n", + " if containers:\n", + " container=containers[0]\n", + " container.stop(timeout=120)\n", + "\n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(stop_triton).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 56, + "id": "2dec80ca-7a7c-46a9-97c0-7afb1572f5b9", + "metadata": {}, + "outputs": [], + "source": [ + "spark.stop()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f43118ab-fc0a-4f64-a126-4302e615654a", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "spark-dl-tf", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/conditional_generation.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/conditional_generation_torch.ipynb similarity index 53% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/conditional_generation.ipynb rename to examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/conditional_generation_torch.ipynb index d3949f846..a09bede39 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/conditional_generation.ipynb +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/conditional_generation_torch.ipynb @@ -6,42 +6,57 @@ "metadata": {}, "source": [ "# PySpark Huggingface Inferencing\n", - "## Conditional generation\n", + "## Conditional generation with PyTorch\n", "\n", "From: https://huggingface.co/docs/transformers/model_doc/t5" ] }, - { - "cell_type": "markdown", - "id": "6bcbccb1-3f82-425b-a82d-12d4f2f91d6e", - "metadata": {}, - "source": [ - "### Using PyTorch" - ] - }, { "cell_type": "code", "execution_count": 1, - "id": "731faab7-a700-46f8-bba5-1c8764e5eacb", + "id": "c0eed0e8", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "/home/leey/.pyenv/versions/3.9.10/envs/spark_rapids_examples/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", - " from .autonotebook import tqdm as notebook_tqdm\n" + "You are using the default legacy behaviour of the . This is expected, and simply means that the `legacy` (previous) behavior will be used so nothing changes for you. If you want to use the new behaviour, set `legacy=False`. This should only be set if you understand what it means, and thoroughly read the reason why this was added as explained in https://github.com/huggingface/transformers/pull/24565\n" ] } ], "source": [ - "from transformers import T5Tokenizer, T5ForConditionalGeneration\n", - "\n", - "tokenizer = T5Tokenizer.from_pretrained(\"t5-small\")\n", - "model = T5ForConditionalGeneration.from_pretrained(\"t5-small\")\n", - "\n", - "max_source_length = 512\n", - "max_target_length = 128\n", + "from transformers import T5Tokenizer, T5ForConditionalGeneration" + ] + }, + { + "cell_type": "markdown", + "id": "041ca559", + "metadata": {}, + "source": [ + "Enabling Huggingface tokenizer parallelism so that it is not automatically disabled with Python parallelism. See [this thread](https://github.com/huggingface/transformers/issues/5486) for more info. " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6695a3e5", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "os.environ[\"TOKENIZERS_PARALLELISM\"] = \"true\"" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "900d6506", + "metadata": {}, + "outputs": [], + "source": [ + "tokenizer = T5Tokenizer.from_pretrained(\"google-t5/t5-small\")\n", + "model = T5ForConditionalGeneration.from_pretrained(\"google-t5/t5-small\")\n", "\n", "task_prefix = \"translate English to German: \"\n", "\n", @@ -57,30 +72,23 @@ { "cell_type": "code", "execution_count": 2, - "id": "45abfa26-02da-4d4a-a925-85b387de0ada", + "id": "73655aea", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/home/leey/.pyenv/versions/3.9.10/envs/spark_rapids_examples/lib/python3.9/site-packages/transformers/generation/utils.py:1346: UserWarning: Using `max_length`'s default (20) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.\n", - " warnings.warn(\n" - ] - } - ], + "outputs": [], "source": [ "input_ids = tokenizer(input_sequences, \n", " padding=\"longest\", \n", - " max_length=max_source_length,\n", + " max_length=512,\n", + " truncation=True,\n", " return_tensors=\"pt\").input_ids\n", - "outputs = model.generate(input_ids)" + "\n", + "outputs = model.generate(input_ids, max_length=20)" ] }, { "cell_type": "code", "execution_count": 3, - "id": "72972ade-3a97-4bb3-9efa-31039a1a9442", + "id": "90e54262", "metadata": {}, "outputs": [ { @@ -103,7 +111,7 @@ { "cell_type": "code", "execution_count": 4, - "id": "18d060a1-19ef-4101-a9a6-2fdc184e07b0", + "id": "6b11c89a", "metadata": {}, "outputs": [ { @@ -123,159 +131,94 @@ }, { "cell_type": "markdown", - "id": "05c79ac4-bf25-421e-b55e-020d6d9e15d5", + "id": "546eabe0", "metadata": {}, "source": [ - "### Using TensorFlow" + "## PySpark" ] }, { "cell_type": "code", - "execution_count": 5, - "id": "f6f0dbf3-712b-4c58-85eb-261ce15bb2be", + "execution_count": 1, + "id": "2f6db1f0-7d68-4af7-8bd6-c9fa45906c61", "metadata": {}, "outputs": [], "source": [ - "from transformers import T5Tokenizer, TFT5ForConditionalGeneration" + "import os\n", + "from pathlib import Path\n", + "from datasets import load_dataset" ] }, { "cell_type": "code", - "execution_count": 6, - "id": "2684fb41-9467-40c0-9d7e-a1cc867c5a3c", + "execution_count": 2, + "id": "68121304-f1df-466e-9347-c9d2b36a9b3a", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "All model checkpoint layers were used when initializing TFT5ForConditionalGeneration.\n", - "\n", - "All the layers of TFT5ForConditionalGeneration were initialized from the model checkpoint at t5-small.\n", - "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFT5ForConditionalGeneration for predictions without further training.\n" - ] - } - ], + "outputs": [], "source": [ - "tokenizer = T5Tokenizer.from_pretrained(\"t5-small\")\n", - "model = TFT5ForConditionalGeneration.from_pretrained(\"t5-small\")\n", - "\n", - "max_source_length = 512\n", - "max_target_length = 128\n", - "\n", - "task_prefix = \"translate English to German: \"\n", - "\n", - "lines = [\n", - " \"The house is wonderful\",\n", - " \"Welcome to NYC\",\n", - " \"HuggingFace is a company\"\n", - "]\n", - "\n", - "input_sequences = [task_prefix + l for l in lines]" + "from pyspark.sql.types import *\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf" ] }, { "cell_type": "code", - "execution_count": 7, - "id": "6eb2dfdb-0ad3-4d0f-81a4-268d92c53759", + "execution_count": 3, + "id": "6279a849", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "/home/leey/.pyenv/versions/3.9.10/envs/spark_rapids_examples/lib/python3.9/site-packages/transformers/generation/tf_utils.py:854: UserWarning: Using `max_length`'s default (20) to control the generation length. This behaviour is deprecated and will be removed from the config in v5 of Transformers -- we recommend using `max_new_tokens` to control the maximum length of the generation.\n", - " warnings.warn(\n" + "24/10/10 00:10:48 WARN Utils: Your hostname, cb4ae00-lcedt resolves to a loopback address: 127.0.1.1; using 10.110.47.100 instead (on interface eno1)\n", + "24/10/10 00:10:48 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n", + "Setting default log level to \"WARN\".\n", + "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n", + "24/10/10 00:10:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n" ] } ], - "source": [ - "input_ids = tokenizer(input_sequences, \n", - " padding=\"longest\", \n", - " max_length=max_source_length,\n", - " return_tensors=\"tf\").input_ids\n", - "outputs = model.generate(input_ids)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "720158d4-e0e0-4904-b096-e5aede756afd", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['Das Haus ist wunderbar',\n", - " 'Willkommen in NYC',\n", - " 'HuggingFace ist ein Unternehmen']" - ] - }, - "execution_count": 8, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "[tokenizer.decode(o, skip_special_tokens=True) for o in outputs]" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "8d4b364b-13cb-48ea-a97a-ccfc9e408075", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'tf'" - ] - }, - "execution_count": 9, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "model.framework" - ] - }, - { - "cell_type": "markdown", - "id": "546eabe0", - "metadata": {}, - "source": [ - "## PySpark" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "2f6db1f0-7d68-4af7-8bd6-c9fa45906c61", - "metadata": {}, - "outputs": [], "source": [ "import os\n", - "from pathlib import Path\n", - "from torchtext.datasets import IMDB" + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.maxRecordsPerBatch\", \"512\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" ] }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 4, "id": "b8453111-d068-49bb-ab91-8ae3d8bcdb7a", "metadata": {}, "outputs": [], "source": [ "# load IMDB reviews (test) dataset\n", - "data = IMDB(split='test')" + "data = load_dataset(\"imdb\", split=\"test\")" ] }, { "cell_type": "code", - "execution_count": 12, - "id": "6d5bb49d-9a5b-4d1c-949e-24d01a7cd9a5", + "execution_count": 5, + "id": "7ad01d4a", "metadata": {}, "outputs": [ { @@ -284,17 +227,16 @@ "25000" ] }, - "execution_count": 12, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "# convert to nested array of string for pyspark\n", "lines = []\n", - "for label, text in data:\n", - " # only take first sentence of IMDB review\n", - " lines.append([text])\n", + "for example in data:\n", + " lines.append([example[\"text\"].split(\".\")[0]])\n", + "\n", "len(lines)" ] }, @@ -308,17 +250,7 @@ }, { "cell_type": "code", - "execution_count": 13, - "id": "68121304-f1df-466e-9347-c9d2b36a9b3a", - "metadata": {}, - "outputs": [], - "source": [ - "from pyspark.sql.types import *" - ] - }, - { - "cell_type": "code", - "execution_count": 14, + "execution_count": 6, "id": "d24d9404-0269-476e-a9dd-1842667c915a", "metadata": {}, "outputs": [ @@ -328,19 +260,19 @@ "StructType([StructField('lines', StringType(), True)])" ] }, - "execution_count": 14, + "execution_count": 6, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "df = spark.createDataFrame(lines, ['lines']).repartition(10)\n", + "df = spark.createDataFrame(lines, ['lines']).repartition(8)\n", "df.schema" ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 7, "id": "4384c762-1f79-4f60-876c-94b1f552e8fb", "metadata": {}, "outputs": [ @@ -348,17 +280,16 @@ "name": "stderr", "output_type": "stream", "text": [ - "23/05/19 19:03:36 WARN TaskSetManager: Stage 0 contains a task of very large size (3858 KiB). The maximum recommended task size is 1000 KiB.\n", " \r" ] }, { "data": { "text/plain": [ - "[Row(lines='Now and again, a film comes around purely by accident that makes you doubt your sanity. We just finished studying the novel, \"Northanger Abbey\", at school and decided to refresh our memory of this unexciting piece of humourless garbage with the BBC adaptation.

The funny thing about Northanger Abbey is that it actually makes you want to kill yourself. The film is NOTHING like the book, for example, the subtly evil characters seem to have been turned into transparent stereotypes. John Thorpe looks like a leprechaun on acid while Isabella plays the role of slut. Catherine, the main character, is the most depressingly stupid and irritating actress on god\\'s earth (she looks like a coffee addict, her eyes are like basketballs) whilst Mr Tilney looks and acts like a retired porno stunt double. The plot goes completely off the rails at certain points of the film, I don\\'t know what the hell the director was thinking when for no reason at all, a 7 year old black kid who we\\'ve never met before takes the main character out of the abbey and starts cartwheeling in front of her. Yes, that\\'s right, cartwheeling. Nonsense of this kind is occasionally interrupted by Catherines \"fantasies\" in which she is being carried around a cathedral by an ogre.

Northanger Abbey is basically visual euthanasia so if you want to murder your boss or something like that, BBC have basically discovered a new way to kill someone. Northanger is a barely laughably bad film. Don\\'t watch it unless you\\'re in a padded cell.')]" + "[Row(lines='(Some Spoilers) Dull as dishwater slasher flick that has this deranged homeless man Harry, Darwyn Swalve, out murdering real-estate agent all over the city of L')]" ] }, - "execution_count": 15, + "execution_count": 7, "metadata": {}, "output_type": "execute_result" } @@ -377,19 +308,10 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 8, "id": "e7eec8ec-4126-4890-b957-025809fad67d", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "23/05/19 19:03:39 WARN TaskSetManager: Stage 3 contains a task of very large size (3858 KiB). The maximum recommended task size is 1000 KiB.\n", - " \r" - ] - } - ], + "outputs": [], "source": [ "df.write.mode(\"overwrite\").parquet(\"imdb_test\")" ] @@ -404,21 +326,14 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 9, "id": "20554ea5-01be-4a30-8607-db5d87786fec", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "23/05/19 19:03:40 WARN TaskSetManager: Stage 6 contains a task of very large size (3858 KiB). The maximum recommended task size is 1000 KiB.\n" - ] - } - ], + "outputs": [], "source": [ - "spark.conf.set(\"spark.sql.execution.arrow.maxRecordsPerBatch\", \"512\")\n", - "# This line will fail if the vectorized reader runs out of memory\n", + "if int(spark.conf.get(\"spark.sql.execution.arrow.maxRecordsPerBatch\")) > 512:\n", + " print(\"Decreasing `spark.sql.execution.arrow.maxRecordsPerBatch` to ensure the vectorized reader won't run out of memory\")\n", + " spark.conf.set(\"spark.sql.execution.arrow.maxRecordsPerBatch\", \"512\")\n", "assert len(df.head()) > 0, \"`df` should not be empty\"" ] }, @@ -427,13 +342,13 @@ "id": "06a4ecab-c9d9-466f-ba49-902ad1fd5488", "metadata": {}, "source": [ - "## Inference using Spark DL API (PyTorch)\n", + "## Inference using Spark DL API\n", "Note: you can restart the kernel and run from this point to simulate running in a different node or environment." ] }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 10, "id": "e7a00479-1347-4de8-8431-faa77f8cdf4c", "metadata": { "tags": [] @@ -448,7 +363,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 11, "id": "b9a0889a-35b4-493a-8197-1146fc7efd53", "metadata": {}, "outputs": [], @@ -463,7 +378,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 12, "id": "c483e4d4-9ab1-416f-a766-694e17490fd3", "metadata": {}, "outputs": [ @@ -474,26 +389,26 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| lines|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "|...But not this one! I always wanted to know \"what happened\" next. We will never know for sure what happened because ...|\n", - "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story without the gratuitous sex scen...|\n", - "|I watched this movie to see the direction one of the most promising young talents in movies was going. Unfortunately,...|\n", - "|This movie makes you wish imdb would let you vote a zero. One of the two movies I've ever walked out of. It's very ha...|\n", - "|I never want to see this movie again!

Not only is it dreadfully bad, but I can't stand seeing my hero Stan...|\n", - "|(As a note, I'd like to say that I saw this movie at my annual church camp, where the entire youth group laughed at i...|\n", - "|Don't get me wrong, I love the TV series of League Of Gentlemen. It was funny, twisted and completely inspired. I was...|\n", - "|Did you ever think, like after watching a horror movie with a group of friends: \"Wow, this is so cool! We have got to...|\n", - "|Awful, awful, awful...

I loved the original film. It was funny, charming, and had heart... this piece of j...|\n", - "|This movie seems a little clunky around the edges, like not quite enough zaniness was thrown it when it should have b...|\n", - "|I rented this movie hoping that it would provide some good entertainment and some cool poker knowledge or stories. Wh...|\n", - "|Well, where to start describing this celluloid debacle? You already know the big fat NADA passing as a plot, so let's...|\n", - "|I hoped for this show to be somewhat realistic. It stroke me as just another mainstream show after I watched it. I di...|\n", - "|All I have to say is one word...SUCKS!!!!. The only reason I gave this a 2 is because Josh Hartnett was in it and he'...|\n", - "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay.

The lighting at is s...|\n", - "|This critique tells the story of 4 little friends who went to watch Angels and Demons the movie on the first night it...|\n", - "|This review contains a partial spoiler.

Shallow from the outset, 'D.O.A.' at least starts as if it might b...|\n", - "|I'm rather surprised that anybody found this film touching or moving.

The basic premise of the film sounde...|\n", - "|If you like bad movies (and you must to watch this one) here's a good one. Not quite as funny as the first, but much ...|\n", - "|This is really bad, the characters were bland, the story was boring, and there is no sex scene. Furthermore, it lacks...|\n", + "| This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| I am a big fan of The ABC Movies of the Week genre|\n", + "|In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Full House\" and the long-defunc...|\n", + "|When The Spirits Within was released, all you heard from Final Fantasy fans was how awful the movie was because it di...|\n", + "| I like to think of myself as a bad movie connoisseur|\n", + "|This film did well at the box office, and the producers of this mess thought the stars had such good chemistry in thi...|\n", + "|Following the pleasingly atmospheric original and the amusingly silly second one, this incredibly dull, slow, and une...|\n", + "| I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| I have read all of the reviews for this direct to video movie|\n", + "|Yes, it was an awful movie, but there was a song near the beginning of the movie, I think, called \"I got a Woody\" or ...|\n", + "| This was the most uninteresting horror flick I have seen to date|\n", + "|I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'The French Connection\", but it...|\n", + "|Heart of Darkness Movie Review Could a book that is well known for its eloquent wording and complicated concepts ever...|\n", + "| A bad movie ABOUT a bad movie|\n", + "|Apart from the fact that this film was made ( I suppose it seemed a good idea at the time considering BOTTOM was so p...|\n", + "|Watching this movie, you just have to ask: What were they thinking? There are so many noticeably bad parts of this mo...|\n", + "| OK, lets start with the best|\n", + "| Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 years|\n", + "| C|\n", + "| Tom and Jerry are transporting goods via airplane to Africa|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" @@ -505,7 +420,7 @@ "100" ] }, - "execution_count": 20, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } @@ -519,7 +434,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 13, "id": "831bc52c-a5c6-4c29-a6da-0566b5167773", "metadata": {}, "outputs": [], @@ -530,24 +445,17 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 14, "id": "46dac59c-5a54-4576-91e0-279c8b375b95", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, { "data": { "text/plain": [ "100" ] }, - "execution_count": 22, + "execution_count": 14, "metadata": {}, "output_type": "execute_result" } @@ -558,7 +466,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 15, "id": "fef1d846-5852-4762-8527-602f32c0d7cd", "metadata": {}, "outputs": [ @@ -569,26 +477,26 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| input|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "| Translate English to German: |\n", - "|Translate English to German: Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story wi...|\n", - "|Translate English to German: I watched this movie to see the direction one of the most promising young talents in mov...|\n", - "| Translate English to German: This movie makes you wish imdb would let you vote a zero|\n", - "|Translate English to German: I never want to see this movie again!

Not only is it dreadfully bad, but I ca...|\n", - "|Translate English to German: (As a note, I'd like to say that I saw this movie at my annual church camp, where the en...|\n", - "| Translate English to German: Don't get me wrong, I love the TV series of League Of Gentlemen|\n", - "|Translate English to German: Did you ever think, like after watching a horror movie with a group of friends: \"Wow, th...|\n", - "| Translate English to German: Awful, awful, awful|\n", - "|Translate English to German: This movie seems a little clunky around the edges, like not quite enough zaniness was th...|\n", - "|Translate English to German: I rented this movie hoping that it would provide some good entertainment and some cool p...|\n", - "|Translate English to German: Well, where to start describing this celluloid debacle? You already know the big fat NAD...|\n", - "| Translate English to German: I hoped for this show to be somewhat realistic|\n", - "| Translate English to German: All I have to say is one word|\n", - "| Translate English to German: Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay|\n", - "|Translate English to German: This critique tells the story of 4 little friends who went to watch Angels and Demons th...|\n", - "| Translate English to German: This review contains a partial spoiler|\n", - "| Translate English to German: I'm rather surprised that anybody found this film touching or moving|\n", - "| Translate English to German: If you like bad movies (and you must to watch this one) here's a good one|\n", - "|Translate English to German: This is really bad, the characters were bland, the story was boring, and there is no sex...|\n", + "| Translate English to German: This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| Translate English to German: I am a big fan of The ABC Movies of the Week genre|\n", + "|Translate English to German: In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Fu...|\n", + "|Translate English to German: When The Spirits Within was released, all you heard from Final Fantasy fans was how awfu...|\n", + "| Translate English to German: I like to think of myself as a bad movie connoisseur|\n", + "|Translate English to German: This film did well at the box office, and the producers of this mess thought the stars h...|\n", + "|Translate English to German: Following the pleasingly atmospheric original and the amusingly silly second one, this i...|\n", + "| Translate English to German: I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| Translate English to German: I have read all of the reviews for this direct to video movie|\n", + "|Translate English to German: Yes, it was an awful movie, but there was a song near the beginning of the movie, I thin...|\n", + "| Translate English to German: This was the most uninteresting horror flick I have seen to date|\n", + "|Translate English to German: I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'T...|\n", + "|Translate English to German: Heart of Darkness Movie Review Could a book that is well known for its eloquent wording ...|\n", + "| Translate English to German: A bad movie ABOUT a bad movie|\n", + "|Translate English to German: Apart from the fact that this film was made ( I suppose it seemed a good idea at the tim...|\n", + "|Translate English to German: Watching this movie, you just have to ask: What were they thinking? There are so many no...|\n", + "| Translate English to German: OK, lets start with the best|\n", + "|Translate English to German: Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 y...|\n", + "| Translate English to German: C|\n", + "| Translate English to German: Tom and Jerry are transporting goods via airplane to Africa|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" @@ -601,7 +509,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 16, "id": "e7ae69d3-70c2-4765-928f-c96a7ba59829", "metadata": {}, "outputs": [], @@ -609,6 +517,7 @@ "def predict_batch_fn():\n", " import numpy as np\n", " from transformers import T5ForConditionalGeneration, T5Tokenizer\n", + "\n", " model = T5ForConditionalGeneration.from_pretrained(\"t5-small\")\n", " tokenizer = T5Tokenizer.from_pretrained(\"t5-small\")\n", "\n", @@ -617,10 +526,12 @@ " input_ids = tokenizer(flattened, \n", " padding=\"longest\", \n", " max_length=128,\n", + " truncation=True,\n", " return_tensors=\"pt\").input_ids\n", - " output_ids = model.generate(input_ids)\n", + " output_ids = model.generate(input_ids, max_length=20)\n", " string_outputs = np.array([tokenizer.decode(o, skip_special_tokens=True) for o in output_ids])\n", " print(\"predict: {}\".format(len(flattened)))\n", + " \n", " return string_outputs\n", " \n", " return predict" @@ -628,7 +539,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 17, "id": "36684f59-d947-43f8-a2e8-c7a423764e88", "metadata": {}, "outputs": [], @@ -640,7 +551,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 18, "id": "6a01c855-8fa1-4765-a3a5-2c9dd872df10", "metadata": {}, "outputs": [ @@ -655,8 +566,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 23.1 ms, sys: 4.47 ms, total: 27.6 ms\n", - "Wall time: 22.6 s\n" + "CPU times: user 6.58 ms, sys: 4.68 ms, total: 11.3 ms\n", + "Wall time: 7.41 s\n" ] }, { @@ -676,7 +587,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 19, "id": "d912d4b0-cd0b-44ea-859a-b23455cc2700", "metadata": {}, "outputs": [ @@ -691,8 +602,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 15.7 ms, sys: 2.96 ms, total: 18.7 ms\n", - "Wall time: 16 s\n" + "CPU times: user 1.87 ms, sys: 1.8 ms, total: 3.67 ms\n", + "Wall time: 5.71 s\n" ] }, { @@ -711,7 +622,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 20, "id": "5fe3d88b-30f7-468f-8db8-1f4118d0f26c", "metadata": {}, "outputs": [ @@ -726,8 +637,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 9.34 ms, sys: 4.6 ms, total: 13.9 ms\n", - "Wall time: 16 s\n" + "CPU times: user 2.99 ms, sys: 1.42 ms, total: 4.42 ms\n", + "Wall time: 5.69 s\n" ] }, { @@ -746,7 +657,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 21, "id": "4ad9b365-4b9a-438e-8fdf-47da55cb1cf4", "metadata": {}, "outputs": [ @@ -764,26 +675,26 @@ "+------------------------------------------------------------+------------------------------------------------------------+\n", "| input| preds|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", - "| Translate English to German: | Übersetzen Sie Englisch.|\n", - "|Translate English to German: Hard up, No proper jobs goin...| Warum nicht die Kinder mieten?|\n", - "|Translate English to German: I watched this movie to see ...|Ich habe diesen Film gesehen, um zu sehen, in welche Rich...|\n", - "|Translate English to German: This movie makes you wish im...|Dieser Film macht Sie sich wünschen, dass imdb Sie es Ihn...|\n", - "|Translate English to German: I never want to see this mov...| Ich möchte diesen Film nie wieder sehen!br />br|\n", - "|Translate English to German: (As a note, I'd like to say ...| (Als eine Bemerkung, möchte ich sagen, dass ich diesen|\n", - "|Translate English to German: Don't get me wrong, I love t...|Verstehen Sie mich nicht falsch, ich liebe die TV-Serie L...|\n", - "|Translate English to German: Did you ever think, like aft...|Haben Sie jemals glaubt, wie nach einem Horrorfilm mit ei...|\n", - "| Translate English to German: Awful, awful, awful| Wüstenlos, schrecklich, schrecklich|\n", - "|Translate English to German: This movie seems a little cl...|Dieser Film scheint etwas schlank um die Ecken zu sein, w...|\n", - "|Translate English to German: I rented this movie hoping t...|Ich miete diesen Film in der Hoffnung, dass er einige gut...|\n", - "|Translate English to German: Well, where to start describ...| Wie kann man dieses Celluloid-Debakel beschreiben?|\n", - "|Translate English to German: I hoped for this show to be ...| Ich hoffe, dass diese Show etwas realistisch sein wird.|\n", - "| Translate English to German: All I have to say is one word| Ich muss nur ein Wort sagen|\n", - "|Translate English to German: Honestly awful film, bad edi...|Honestly awful film, bad editing, awful lighting, dire di...|\n", - "|Translate English to German: This critique tells the stor...|Diese Kritik erzählt die Geschichte von 4 kleinen Freunde...|\n", - "|Translate English to German: This review contains a parti...| Dieses Review enthält einen Teil-Störer|\n", - "|Translate English to German: I'm rather surprised that an...|Ich bin ziemlich überrascht, dass jemand diesen Film berü...|\n", - "|Translate English to German: If you like bad movies (and ...|Wenn Sie schlechte Filme (und Sie müssen diesen schauen) ...|\n", - "|Translate English to German: This is really bad, the char...|Das ist wirklich schlecht, die Charaktere waren schmeiche...|\n", + "|Translate English to German: This is so overly clichéd yo...| Das ist so übertrieben klischeehaft, dass Sie es nach den|\n", + "|Translate English to German: I am a big fan of The ABC Mo...| Ich bin ein großer Fan von The ABC Movies of the Week|\n", + "|Translate English to German: In the early 1990's \"Step-by...| Anfang der 1990er Jahre kam \"Step-by-Step\" als müh|\n", + "|Translate English to German: When The Spirits Within was ...|Als The Spirits Within veröffentlicht wurde, hörten Sie v...|\n", + "|Translate English to German: I like to think of myself as...| Ich halte mich gerne als schlechter Filmliebhaber|\n", + "|Translate English to German: This film did well at the bo...|Dieser Film hat sich gut an der Boxoffice ereignet, und d...|\n", + "|Translate English to German: Following the pleasingly atm...|Nach dem erfreulich stimmungsvollen Original und dem amüsant|\n", + "|Translate English to German: I like CKY and Viva La Bam, ...| Ich mag CKY und Viva La Bam, also konnte ich mich nicht|\n", + "|Translate English to German: I have read all of the revie...| Ich habe alle Rezensionen zu diesem direkten Film gelesen.|\n", + "|Translate English to German: Yes, it was an awful movie, ...| Ja, es war ein schrecklicher Film, aber es gab|\n", + "|Translate English to German: This was the most uninterest...|Dies war der größte Horrorfilm, den ich bisher gesehen habe.|\n", + "|Translate English to German: I don't know if this excepti...|Ich weiß nicht, ob dieser außergewöhnlich langweilige Fil...|\n", + "|Translate English to German: Heart of Darkness Movie Revi...|Herz der Dunkelheit Film Review Kann ein Buch, das für se...|\n", + "| Translate English to German: A bad movie ABOUT a bad movie| Ein schlechter Film ABOUT a bad movie|\n", + "|Translate English to German: Apart from the fact that thi...| Dieser Film wurde zwar fertiggestellt, aber es schien mir|\n", + "|Translate English to German: Watching this movie, you jus...|Wenn man diesen Film anschaut, muss man einfach fragen: W...|\n", + "| Translate English to German: OK, lets start with the best| OK, lets start with the best|\n", + "|Translate English to German: Anna Christie (Greta Garbo) ...| Anna Christie (Greta Garbo) kehrt nach 15 Jahren zurück,|\n", + "| Translate English to German: C| C|\n", + "|Translate English to German: Tom and Jerry are transporti...|Tom und Jerry transportieren Güter über Flugzeug nach Afrika|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" @@ -803,7 +714,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 22, "id": "1eb0c83b-d91b-4f8c-a5e7-c35f55c88108", "metadata": {}, "outputs": [], @@ -814,17 +725,10 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 23, "id": "054f94fd-fe79-41e7-b1c7-6124083acc72", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[Stage 28:=============================================> (8 + 2) / 10]\r" - ] - }, { "name": "stdout", "output_type": "stream", @@ -832,37 +736,30 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| input|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "| Translate English to French: |\n", - "|Translate English to French: Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story wi...|\n", - "|Translate English to French: I watched this movie to see the direction one of the most promising young talents in mov...|\n", - "| Translate English to French: This movie makes you wish imdb would let you vote a zero|\n", - "|Translate English to French: I never want to see this movie again!

Not only is it dreadfully bad, but I ca...|\n", - "|Translate English to French: (As a note, I'd like to say that I saw this movie at my annual church camp, where the en...|\n", - "| Translate English to French: Don't get me wrong, I love the TV series of League Of Gentlemen|\n", - "|Translate English to French: Did you ever think, like after watching a horror movie with a group of friends: \"Wow, th...|\n", - "| Translate English to French: Awful, awful, awful|\n", - "|Translate English to French: This movie seems a little clunky around the edges, like not quite enough zaniness was th...|\n", - "|Translate English to French: I rented this movie hoping that it would provide some good entertainment and some cool p...|\n", - "|Translate English to French: Well, where to start describing this celluloid debacle? You already know the big fat NAD...|\n", - "| Translate English to French: I hoped for this show to be somewhat realistic|\n", - "| Translate English to French: All I have to say is one word|\n", - "| Translate English to French: Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay|\n", - "|Translate English to French: This critique tells the story of 4 little friends who went to watch Angels and Demons th...|\n", - "| Translate English to French: This review contains a partial spoiler|\n", - "| Translate English to French: I'm rather surprised that anybody found this film touching or moving|\n", - "| Translate English to French: If you like bad movies (and you must to watch this one) here's a good one|\n", - "|Translate English to French: This is really bad, the characters were bland, the story was boring, and there is no sex...|\n", + "| Translate English to French: This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| Translate English to French: I am a big fan of The ABC Movies of the Week genre|\n", + "|Translate English to French: In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Fu...|\n", + "|Translate English to French: When The Spirits Within was released, all you heard from Final Fantasy fans was how awfu...|\n", + "| Translate English to French: I like to think of myself as a bad movie connoisseur|\n", + "|Translate English to French: This film did well at the box office, and the producers of this mess thought the stars h...|\n", + "|Translate English to French: Following the pleasingly atmospheric original and the amusingly silly second one, this i...|\n", + "| Translate English to French: I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| Translate English to French: I have read all of the reviews for this direct to video movie|\n", + "|Translate English to French: Yes, it was an awful movie, but there was a song near the beginning of the movie, I thin...|\n", + "| Translate English to French: This was the most uninteresting horror flick I have seen to date|\n", + "|Translate English to French: I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'T...|\n", + "|Translate English to French: Heart of Darkness Movie Review Could a book that is well known for its eloquent wording ...|\n", + "| Translate English to French: A bad movie ABOUT a bad movie|\n", + "|Translate English to French: Apart from the fact that this film was made ( I suppose it seemed a good idea at the tim...|\n", + "|Translate English to French: Watching this movie, you just have to ask: What were they thinking? There are so many no...|\n", + "| Translate English to French: OK, lets start with the best|\n", + "|Translate English to French: Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 y...|\n", + "| Translate English to French: C|\n", + "| Translate English to French: Tom and Jerry are transporting goods via airplane to Africa|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] } ], "source": [ @@ -871,7 +768,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 24, "id": "6f6b70f9-188a-402b-9143-78a5788140e4", "metadata": {}, "outputs": [ @@ -879,15 +776,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 31:> (0 + 1) / 1]\r" + "[Stage 33:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 20.3 ms, sys: 0 ns, total: 20.3 ms\n", - "Wall time: 22 s\n" + "CPU times: user 2.46 ms, sys: 2.2 ms, total: 4.67 ms\n", + "Wall time: 7.38 s\n" ] }, { @@ -907,7 +804,7 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 25, "id": "031a6a5e-7999-4653-b394-19ed478d8c96", "metadata": {}, "outputs": [ @@ -915,15 +812,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 33:> (0 + 1) / 1]\r" + "[Stage 35:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 16.8 ms, sys: 0 ns, total: 16.8 ms\n", - "Wall time: 15.4 s\n" + "CPU times: user 3.34 ms, sys: 1.13 ms, total: 4.47 ms\n", + "Wall time: 6.1 s\n" ] }, { @@ -942,7 +839,7 @@ }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 26, "id": "229b6515-82f6-4e9c-90f0-a9c3cfb26301", "metadata": {}, "outputs": [ @@ -950,15 +847,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 35:> (0 + 1) / 1]\r" + "[Stage 37:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 12.6 ms, sys: 791 µs, total: 13.3 ms\n", - "Wall time: 15.4 s\n" + "CPU times: user 1.72 ms, sys: 2.89 ms, total: 4.6 ms\n", + "Wall time: 5.93 s\n" ] }, { @@ -977,7 +874,7 @@ }, { "cell_type": "code", - "execution_count": 35, + "execution_count": 27, "id": "8be750ac-fa39-452e-bb4c-c2270bc2f70d", "metadata": {}, "outputs": [ @@ -985,7 +882,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 37:> (0 + 1) / 1]\r" + "[Stage 39:> (0 + 1) / 1]\r" ] }, { @@ -995,26 +892,26 @@ "+------------------------------------------------------------+------------------------------------------------------------+\n", "| input| preds|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", - "| Translate English to French: | :|\n", - "|Translate English to French: Hard up, No proper jobs goin...| Vous ne pouvez pas louer vos enfants!|\n", - "|Translate English to French: I watched this movie to see ...|J’ai regardé ce film pour voir la direction d’un des jeun...|\n", - "|Translate English to French: This movie makes you wish im...|Ce film vous fait envie de voir imdb vous laisser voter zéro|\n", - "|Translate English to French: I never want to see this mov...| Je ne veux jamais voir ce film à nouveau!br /|\n", - "|Translate English to French: (As a note, I'd like to say ...| ( titre de note, je voudrais dire que j'ai vu ce film|\n", - "|Translate English to French: Don't get me wrong, I love t...|Ne m'oubliez pas, je m'aime la série de télévision de League|\n", - "|Translate English to French: Did you ever think, like aft...|Vous avez jamais pensé, comme après avoir vu un film d'horre|\n", - "| Translate English to French: Awful, awful, awful| Awful, awful, awful|\n", - "|Translate English to French: This movie seems a little cl...| Ce film semble un peu cloné autour des bords, comme il|\n", - "|Translate English to French: I rented this movie hoping t...| J'ai loué ce film en espérant qu'il fournirait|\n", - "|Translate English to French: Well, where to start describ...|Vous savez déjà que la grande graisse NADA passe comme un...|\n", - "|Translate English to French: I hoped for this show to be ...| J'espère que ce spectacle sera quelque peu réaliste|\n", - "| Translate English to French: All I have to say is one word| Je n'ai qu'à dire un mot|\n", - "|Translate English to French: Honestly awful film, bad edi...| l'instar de l'arrière-plan, il|\n", - "|Translate English to French: This critique tells the stor...|Cette critique raconte l'histoire de 4 petits amis qui on...|\n", - "|Translate English to French: This review contains a parti...| Cet examen contient un spoiler partiel|\n", - "|Translate English to French: I'm rather surprised that an...|Je suis plutôt surpris que quelqu'un ait trouvé ce film t...|\n", - "|Translate English to French: If you like bad movies (and ...|Si vous aimez des films mauvais (et vous devez regarder c...|\n", - "|Translate English to French: This is really bad, the char...| C'est vraiment mauvais, les personnages étaient bourds, l|\n", + "|Translate English to French: This is so overly clichéd yo...| Vous ne pouvez pas en tirer d'un tel cliché|\n", + "|Translate English to French: I am a big fan of The ABC Mo...| Je suis un grand fan du genre The ABC Movies of the Week|\n", + "|Translate English to French: In the early 1990's \"Step-by...| Au début des années 1990, «Step-by-Step» a été une|\n", + "|Translate English to French: When The Spirits Within was ...|Lorsque The Spirits Within a été publié, tout ce que vous...|\n", + "|Translate English to French: I like to think of myself as...| Je me considère comme un mauvais réalisateur de films|\n", + "|Translate English to French: This film did well at the bo...| Ce film a bien avancé à la salle de cinéma et|\n", + "|Translate English to French: Following the pleasingly atm...|Après l'original agréablement atmosphérique et la seconde...|\n", + "|Translate English to French: I like CKY and Viva La Bam, ...| Je m'aime CKY et Viva La Bam,|\n", + "|Translate English to French: I have read all of the revie...| J'ai lu tous les commentaires pour ce film direct à vidéo|\n", + "|Translate English to French: Yes, it was an awful movie, ...| Oui, c'était un film terrible, mais il y avait une chanson|\n", + "|Translate English to French: This was the most uninterest...| Ce fut le plus inquiétant et le plus inquiétant d'h|\n", + "|Translate English to French: I don't know if this excepti...|Je ne sais pas si ce film extrêmement tacheté était desti...|\n", + "|Translate English to French: Heart of Darkness Movie Revi...| Un livre connu pour son éloquence et ses concepts complexes|\n", + "| Translate English to French: A bad movie ABOUT a bad movie| Un mauvais film ABOUT a bad movie|\n", + "|Translate English to French: Apart from the fact that thi...|En plus du fait que ce film a été réalisé (je suppose quil s|\n", + "|Translate English to French: Watching this movie, you jus...|Vous devez simplement vous demander : « Que pense-t-il? » Il|\n", + "| Translate English to French: OK, lets start with the best| OK, s'il y a lieu de commencer par le meilleur|\n", + "|Translate English to French: Anna Christie (Greta Garbo) ...|Anna Christie (Greta Garbo) retourne pour voir son père C...|\n", + "| Translate English to French: C| C|\n", + "|Translate English to French: Tom and Jerry are transporti...| Tom et Jerry transportent des marchandises par avion en|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" @@ -1047,36 +944,31 @@ "id": "5d98fa52-7665-49bf-865a-feec86effe23", "metadata": {}, "source": [ - "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments), using a conda-pack environment created as follows:\n", + "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments) with the compatible versions of Python/Numpy for Triton 24.08, using a conda-pack environment created as follows:\n", "```\n", - "conda create -n huggingface -c conda-forge python=3.8\n", - "conda activate huggingface\n", + "conda create -n huggingface-torch -c conda-forge python=3.10.0\n", + "conda activate huggingface-torch\n", "\n", - "export PYTHONUSERSITE=True\n", - "pip install conda-pack sentencepiece sentence_transformers transformers\n", + "export PYTHONNOUSERSITE=True\n", + "pip install numpy==1.26.4 conda-pack sentencepiece sentence_transformers transformers\n", "\n", - "conda-pack # huggingface.tar.gz\n", + "conda-pack # huggingface-torch.tar.gz\n", "```" ] }, { "cell_type": "code", - "execution_count": 36, + "execution_count": 28, "id": "b858cf85-82e6-41ef-905b-d8c5d6fea492", "metadata": {}, "outputs": [], "source": [ - "import numpy as np\n", - "import pandas as pd\n", - "import os\n", - "from pyspark.ml.functions import predict_batch_udf\n", - "from pyspark.sql.functions import col, struct, pandas_udf\n", - "from pyspark.sql.types import FloatType, StringType, StructField, StructType" + "import os" ] }, { "cell_type": "code", - "execution_count": 37, + "execution_count": 29, "id": "05ce7c77-d562-45e8-89bb-cd656aba5a5f", "metadata": {}, "outputs": [], @@ -1085,10 +977,10 @@ "# copy custom model to expected layout for Triton\n", "rm -rf models\n", "mkdir -p models\n", - "cp -r models_config/hf_generation models\n", + "cp -r models_config/hf_generation_torch models\n", "\n", "# add custom execution environment\n", - "cp huggingface.tar.gz models" + "cp huggingface-torch.tar.gz models" ] }, { @@ -1103,7 +995,7 @@ }, { "cell_type": "code", - "execution_count": 38, + "execution_count": 30, "id": "afd00b7e-8150-4c95-a2e4-037e9c90f92a", "metadata": {}, "outputs": [ @@ -1120,7 +1012,7 @@ "[True]" ] }, - "execution_count": 38, + "execution_count": 30, "metadata": {}, "output_type": "execute_result" } @@ -1142,7 +1034,7 @@ " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", " else:\n", " container=client.containers.run(\n", - " \"nvcr.io/nvidia/tritonserver:23.04-py3\", \"tritonserver --model-repository=/models\",\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", " detach=True,\n", " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", " environment=[\n", @@ -1151,7 +1043,7 @@ " name=\"spark-triton\",\n", " network_mode=\"host\",\n", " remove=True,\n", - " shm_size=\"256M\",\n", + " shm_size=\"1G\",\n", " volumes={\n", " triton_models_dir: {\"bind\": \"/models\", \"mode\": \"ro\"},\n", " huggingface_cache_dir: {\"bind\": \"/cache\", \"mode\": \"rw\"}\n", @@ -1184,7 +1076,7 @@ }, { "cell_type": "code", - "execution_count": 39, + "execution_count": 31, "id": "1a997c33-5202-466d-8304-b8c30f32978f", "metadata": { "tags": [] @@ -1200,7 +1092,7 @@ }, { "cell_type": "code", - "execution_count": 40, + "execution_count": 32, "id": "9dea1875-6b95-4fc0-926d-a625a441b33d", "metadata": {}, "outputs": [], @@ -1211,7 +1103,7 @@ }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 33, "id": "5d6c54e7-534d-406f-b8e6-fd592efd0ab2", "metadata": {}, "outputs": [], @@ -1226,7 +1118,7 @@ }, { "cell_type": "code", - "execution_count": 42, + "execution_count": 34, "id": "dc1bbbe3-4232-49e5-80f6-99976524b73b", "metadata": {}, "outputs": [], @@ -1237,7 +1129,7 @@ }, { "cell_type": "code", - "execution_count": 43, + "execution_count": 35, "id": "5d10c61c-6102-4d19-8dd6-0c7b5b65343e", "metadata": {}, "outputs": [ @@ -1248,26 +1140,26 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| input|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "| Translate English to German: |\n", - "|Translate English to German: Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story wi...|\n", - "|Translate English to German: I watched this movie to see the direction one of the most promising young talents in mov...|\n", - "| Translate English to German: This movie makes you wish imdb would let you vote a zero|\n", - "|Translate English to German: I never want to see this movie again!

Not only is it dreadfully bad, but I ca...|\n", - "|Translate English to German: (As a note, I'd like to say that I saw this movie at my annual church camp, where the en...|\n", - "| Translate English to German: Don't get me wrong, I love the TV series of League Of Gentlemen|\n", - "|Translate English to German: Did you ever think, like after watching a horror movie with a group of friends: \"Wow, th...|\n", - "| Translate English to German: Awful, awful, awful|\n", - "|Translate English to German: This movie seems a little clunky around the edges, like not quite enough zaniness was th...|\n", - "|Translate English to German: I rented this movie hoping that it would provide some good entertainment and some cool p...|\n", - "|Translate English to German: Well, where to start describing this celluloid debacle? You already know the big fat NAD...|\n", - "| Translate English to German: I hoped for this show to be somewhat realistic|\n", - "| Translate English to German: All I have to say is one word|\n", - "| Translate English to German: Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay|\n", - "|Translate English to German: This critique tells the story of 4 little friends who went to watch Angels and Demons th...|\n", - "| Translate English to German: This review contains a partial spoiler|\n", - "| Translate English to German: I'm rather surprised that anybody found this film touching or moving|\n", - "| Translate English to German: If you like bad movies (and you must to watch this one) here's a good one|\n", - "|Translate English to German: This is really bad, the characters were bland, the story was boring, and there is no sex...|\n", + "| Translate English to German: This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| Translate English to German: I am a big fan of The ABC Movies of the Week genre|\n", + "|Translate English to German: In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Fu...|\n", + "|Translate English to German: When The Spirits Within was released, all you heard from Final Fantasy fans was how awfu...|\n", + "| Translate English to German: I like to think of myself as a bad movie connoisseur|\n", + "|Translate English to German: This film did well at the box office, and the producers of this mess thought the stars h...|\n", + "|Translate English to German: Following the pleasingly atmospheric original and the amusingly silly second one, this i...|\n", + "| Translate English to German: I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| Translate English to German: I have read all of the reviews for this direct to video movie|\n", + "|Translate English to German: Yes, it was an awful movie, but there was a song near the beginning of the movie, I thin...|\n", + "| Translate English to German: This was the most uninteresting horror flick I have seen to date|\n", + "|Translate English to German: I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'T...|\n", + "|Translate English to German: Heart of Darkness Movie Review Could a book that is well known for its eloquent wording ...|\n", + "| Translate English to German: A bad movie ABOUT a bad movie|\n", + "|Translate English to German: Apart from the fact that this film was made ( I suppose it seemed a good idea at the tim...|\n", + "|Translate English to German: Watching this movie, you just have to ask: What were they thinking? There are so many no...|\n", + "| Translate English to German: OK, lets start with the best|\n", + "|Translate English to German: Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 y...|\n", + "| Translate English to German: C|\n", + "| Translate English to German: Tom and Jerry are transporting goods via airplane to Africa|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" @@ -1280,7 +1172,7 @@ }, { "cell_type": "code", - "execution_count": 44, + "execution_count": 36, "id": "2e0907da-a5d9-4c3b-9db4-ce5e70ca9bb4", "metadata": {}, "outputs": [], @@ -1330,12 +1222,12 @@ }, { "cell_type": "code", - "execution_count": 45, + "execution_count": 37, "id": "9308bdd7-6f67-484d-8b51-dd1e1b2960ba", "metadata": {}, "outputs": [], "source": [ - "generate = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"hf_generation\"),\n", + "generate = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"hf_generation_torch\"),\n", " return_type=StringType(),\n", " input_tensor_shapes=[[1]],\n", " batch_size=100)" @@ -1343,7 +1235,7 @@ }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 38, "id": "38484ffd-370d-492b-8ca4-9eff9f242a9f", "metadata": {}, "outputs": [ @@ -1351,15 +1243,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 43:> (0 + 1) / 1]\r" + "[Stage 45:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 22.3 ms, sys: 4.66 ms, total: 27 ms\n", - "Wall time: 4.47 s\n" + "CPU times: user 4.61 ms, sys: 1.26 ms, total: 5.87 ms\n", + "Wall time: 2.04 s\n" ] }, { @@ -1379,7 +1271,7 @@ }, { "cell_type": "code", - "execution_count": 47, + "execution_count": 39, "id": "ebcb6699-3ac2-4529-ab0f-fab0a5e792da", "metadata": {}, "outputs": [ @@ -1387,15 +1279,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 45:> (0 + 1) / 1]\r" + "[Stage 47:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 9.54 ms, sys: 4.29 ms, total: 13.8 ms\n", - "Wall time: 4.31 s\n" + "CPU times: user 3.16 ms, sys: 641 μs, total: 3.8 ms\n", + "Wall time: 1.58 s\n" ] }, { @@ -1414,7 +1306,7 @@ }, { "cell_type": "code", - "execution_count": 48, + "execution_count": 40, "id": "e2ed18ad-d00b-472c-b2c3-047932f2105d", "metadata": {}, "outputs": [ @@ -1422,15 +1314,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 47:> (0 + 1) / 1]\r" + "[Stage 49:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 11.7 ms, sys: 9.72 ms, total: 21.4 ms\n", - "Wall time: 4.22 s\n" + "CPU times: user 1.91 ms, sys: 2.38 ms, total: 4.29 ms\n", + "Wall time: 1.75 s\n" ] }, { @@ -1449,7 +1341,7 @@ }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 41, "id": "0cd64a1c-beb8-47d5-ac6f-e8525bb61176", "metadata": {}, "outputs": [ @@ -1460,26 +1352,26 @@ "+------------------------------------------------------------+------------------------------------------------------------+\n", "| input| preds|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", - "| Translate English to German: | Übersetzen Sie Englisch.|\n", - "|Translate English to German: Hard up, No proper jobs goin...| Warum nicht die Kinder mieten?|\n", - "|Translate English to German: I watched this movie to see ...|Ich habe diesen Film gesehen, um zu sehen, in welche Rich...|\n", - "|Translate English to German: This movie makes you wish im...|Dieser Film macht Sie sich wünschen, dass imdb Sie es Ihn...|\n", - "|Translate English to German: I never want to see this mov...| Ich möchte diesen Film nie wieder sehen!br />br|\n", - "|Translate English to German: (As a note, I'd like to say ...| (Als eine Bemerkung, möchte ich sagen, dass ich diesen|\n", - "|Translate English to German: Don't get me wrong, I love t...|Verstehen Sie mich nicht falsch, ich liebe die TV-Serie L...|\n", - "|Translate English to German: Did you ever think, like aft...|Haben Sie jemals glaubt, wie nach einem Horrorfilm mit ei...|\n", - "| Translate English to German: Awful, awful, awful| Wüstenlos, schrecklich, schrecklich|\n", - "|Translate English to German: This movie seems a little cl...|Dieser Film scheint etwas schlank um die Ecken zu sein, w...|\n", - "|Translate English to German: I rented this movie hoping t...|Ich miete diesen Film in der Hoffnung, dass er einige gut...|\n", - "|Translate English to German: Well, where to start describ...| Wie kann man dieses Celluloid-Debakel beschreiben?|\n", - "|Translate English to German: I hoped for this show to be ...| Ich hoffe, dass diese Show etwas realistisch sein wird.|\n", - "| Translate English to German: All I have to say is one word| Ich muss nur ein Wort sagen|\n", - "|Translate English to German: Honestly awful film, bad edi...|Honestly awful film, bad editing, awful lighting, dire di...|\n", - "|Translate English to German: This critique tells the stor...|Diese Kritik erzählt die Geschichte von 4 kleinen Freunde...|\n", - "|Translate English to German: This review contains a parti...| Dieses Review enthält einen Teil-Störer|\n", - "|Translate English to German: I'm rather surprised that an...|Ich bin ziemlich überrascht, dass jemand diesen Film berü...|\n", - "|Translate English to German: If you like bad movies (and ...|Wenn Sie schlechte Filme (und Sie müssen diesen schauen) ...|\n", - "|Translate English to German: This is really bad, the char...|Das ist wirklich schlecht, die Charaktere waren schmeiche...|\n", + "|Translate English to German: This is so overly clichéd yo...| Das ist so übertrieben klischeehaft, dass Sie es nach den|\n", + "|Translate English to German: I am a big fan of The ABC Mo...| Ich bin ein großer Fan von The ABC Movies of the Week|\n", + "|Translate English to German: In the early 1990's \"Step-by...| Anfang der 1990er Jahre kam \"Step-by-Step\" als müh|\n", + "|Translate English to German: When The Spirits Within was ...|Als The Spirits Within veröffentlicht wurde, hörten Sie v...|\n", + "|Translate English to German: I like to think of myself as...| Ich halte mich gerne als schlechter Filmliebhaber|\n", + "|Translate English to German: This film did well at the bo...|Dieser Film hat sich gut an der Boxoffice ereignet, und d...|\n", + "|Translate English to German: Following the pleasingly atm...|Nach dem erfreulich stimmungsvollen Original und dem amüsant|\n", + "|Translate English to German: I like CKY and Viva La Bam, ...| Ich mag CKY und Viva La Bam, also konnte ich mich nicht|\n", + "|Translate English to German: I have read all of the revie...| Ich habe alle Rezensionen zu diesem direkten Film gelesen.|\n", + "|Translate English to German: Yes, it was an awful movie, ...| Ja, es war ein schrecklicher Film, aber es gab|\n", + "|Translate English to German: This was the most uninterest...|Dies war der größte Horrorfilm, den ich bisher gesehen habe.|\n", + "|Translate English to German: I don't know if this excepti...|Ich weiß nicht, ob dieser außergewöhnlich langweilige Fil...|\n", + "|Translate English to German: Heart of Darkness Movie Revi...|Herz der Dunkelheit Film Review Kann ein Buch, das für se...|\n", + "| Translate English to German: A bad movie ABOUT a bad movie| Ein schlechter Film ABOUT a bad movie|\n", + "|Translate English to German: Apart from the fact that thi...| Dieser Film wurde zwar fertiggestellt, aber es schien mir|\n", + "|Translate English to German: Watching this movie, you jus...|Wenn man diesen Film anschaut, muss man einfach fragen: W...|\n", + "| Translate English to German: OK, lets start with the best| OK, lets start with the best|\n", + "|Translate English to German: Anna Christie (Greta Garbo) ...| Anna Christie (Greta Garbo) kehrt nach 15 Jahren zurück,|\n", + "| Translate English to German: C| C|\n", + "|Translate English to German: Tom and Jerry are transporti...|Tom und Jerry transportieren Güter über Flugzeug nach Afrika|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" @@ -1492,7 +1384,7 @@ }, { "cell_type": "code", - "execution_count": 50, + "execution_count": 42, "id": "af70fed8-0f2b-4ea7-841c-476afdf9b1c0", "metadata": {}, "outputs": [ @@ -1500,7 +1392,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "23/05/19 19:06:28 WARN CacheManager: Asked to cache already cached data.\n" + "24/10/10 00:12:21 WARN CacheManager: Asked to cache already cached data.\n" ] } ], @@ -1511,7 +1403,7 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 43, "id": "ef075e10-e22c-4236-9e0b-cb47cf2d3d06", "metadata": {}, "outputs": [ @@ -1522,26 +1414,26 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| input|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "| Translate English to French: |\n", - "|Translate English to French: Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story wi...|\n", - "|Translate English to French: I watched this movie to see the direction one of the most promising young talents in mov...|\n", - "| Translate English to French: This movie makes you wish imdb would let you vote a zero|\n", - "|Translate English to French: I never want to see this movie again!

Not only is it dreadfully bad, but I ca...|\n", - "|Translate English to French: (As a note, I'd like to say that I saw this movie at my annual church camp, where the en...|\n", - "| Translate English to French: Don't get me wrong, I love the TV series of League Of Gentlemen|\n", - "|Translate English to French: Did you ever think, like after watching a horror movie with a group of friends: \"Wow, th...|\n", - "| Translate English to French: Awful, awful, awful|\n", - "|Translate English to French: This movie seems a little clunky around the edges, like not quite enough zaniness was th...|\n", - "|Translate English to French: I rented this movie hoping that it would provide some good entertainment and some cool p...|\n", - "|Translate English to French: Well, where to start describing this celluloid debacle? You already know the big fat NAD...|\n", - "| Translate English to French: I hoped for this show to be somewhat realistic|\n", - "| Translate English to French: All I have to say is one word|\n", - "| Translate English to French: Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay|\n", - "|Translate English to French: This critique tells the story of 4 little friends who went to watch Angels and Demons th...|\n", - "| Translate English to French: This review contains a partial spoiler|\n", - "| Translate English to French: I'm rather surprised that anybody found this film touching or moving|\n", - "| Translate English to French: If you like bad movies (and you must to watch this one) here's a good one|\n", - "|Translate English to French: This is really bad, the characters were bland, the story was boring, and there is no sex...|\n", + "| Translate English to French: This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| Translate English to French: I am a big fan of The ABC Movies of the Week genre|\n", + "|Translate English to French: In the early 1990's \"Step-by-Step\" came as a tedious combination of the ultra-cheesy \"Fu...|\n", + "|Translate English to French: When The Spirits Within was released, all you heard from Final Fantasy fans was how awfu...|\n", + "| Translate English to French: I like to think of myself as a bad movie connoisseur|\n", + "|Translate English to French: This film did well at the box office, and the producers of this mess thought the stars h...|\n", + "|Translate English to French: Following the pleasingly atmospheric original and the amusingly silly second one, this i...|\n", + "| Translate English to French: I like CKY and Viva La Bam, so I couldn't resist this when I saw it for £1|\n", + "| Translate English to French: I have read all of the reviews for this direct to video movie|\n", + "|Translate English to French: Yes, it was an awful movie, but there was a song near the beginning of the movie, I thin...|\n", + "| Translate English to French: This was the most uninteresting horror flick I have seen to date|\n", + "|Translate English to French: I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'T...|\n", + "|Translate English to French: Heart of Darkness Movie Review Could a book that is well known for its eloquent wording ...|\n", + "| Translate English to French: A bad movie ABOUT a bad movie|\n", + "|Translate English to French: Apart from the fact that this film was made ( I suppose it seemed a good idea at the tim...|\n", + "|Translate English to French: Watching this movie, you just have to ask: What were they thinking? There are so many no...|\n", + "| Translate English to French: OK, lets start with the best|\n", + "|Translate English to French: Anna Christie (Greta Garbo) returns to see her father Chris (George F Marion) after 15 y...|\n", + "| Translate English to French: C|\n", + "| Translate English to French: Tom and Jerry are transporting goods via airplane to Africa|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" @@ -1554,7 +1446,7 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 44, "id": "2e7e4af8-b815-4375-b851-8368309ee8e1", "metadata": {}, "outputs": [ @@ -1562,15 +1454,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 53:> (0 + 1) / 1]\r" + "[Stage 55:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 8.14 ms, sys: 12.6 ms, total: 20.8 ms\n", - "Wall time: 4.75 s\n" + "CPU times: user 3.4 ms, sys: 2.75 ms, total: 6.14 ms\n", + "Wall time: 1.96 s\n" ] }, { @@ -1589,7 +1481,7 @@ }, { "cell_type": "code", - "execution_count": 53, + "execution_count": 45, "id": "7b0aefb0-a96b-4791-a23c-1ce9b24eb20c", "metadata": {}, "outputs": [ @@ -1597,15 +1489,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 55:> (0 + 1) / 1]\r" + "[Stage 57:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 11.6 ms, sys: 3 ms, total: 14.6 ms\n", - "Wall time: 3.87 s\n" + "CPU times: user 3.76 ms, sys: 897 μs, total: 4.66 ms\n", + "Wall time: 1.61 s\n" ] }, { @@ -1624,7 +1516,7 @@ }, { "cell_type": "code", - "execution_count": 54, + "execution_count": 46, "id": "1214b75b-a373-4579-b4c6-0cb8627da776", "metadata": {}, "outputs": [ @@ -1632,15 +1524,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 57:> (0 + 1) / 1]\r" + "[Stage 59:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 13.1 ms, sys: 4.3 ms, total: 17.4 ms\n", - "Wall time: 3.9 s\n" + "CPU times: user 2.61 ms, sys: 2.26 ms, total: 4.87 ms\n", + "Wall time: 1.67 s\n" ] }, { @@ -1659,7 +1551,7 @@ }, { "cell_type": "code", - "execution_count": 55, + "execution_count": 47, "id": "c9dbd21f-9e37-4221-b765-80ba8c80b884", "metadata": {}, "outputs": [ @@ -1670,37 +1562,30 @@ "+------------------------------------------------------------+------------------------------------------------------------+\n", "| input| preds|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", - "| Translate English to French: | :|\n", - "|Translate English to French: Hard up, No proper jobs goin...| Vous ne pouvez pas louer vos enfants!|\n", - "|Translate English to French: I watched this movie to see ...|J’ai regardé ce film pour voir la direction d’un des jeun...|\n", - "|Translate English to French: This movie makes you wish im...|Ce film vous fait envie de voir imdb vous laisser voter zéro|\n", - "|Translate English to French: I never want to see this mov...| Je ne veux jamais voir ce film à nouveau!br /|\n", - "|Translate English to French: (As a note, I'd like to say ...| ( titre de note, je voudrais dire que j'ai vu ce film|\n", - "|Translate English to French: Don't get me wrong, I love t...|Ne m'oubliez pas, je m'aime la série de télévision de League|\n", - "|Translate English to French: Did you ever think, like aft...|Vous avez jamais pensé, comme après avoir vu un film d'horre|\n", - "| Translate English to French: Awful, awful, awful| Awful, awful, awful|\n", - "|Translate English to French: This movie seems a little cl...| Ce film semble un peu cloné autour des bords, comme il|\n", - "|Translate English to French: I rented this movie hoping t...| J'ai loué ce film en espérant qu'il fournirait|\n", - "|Translate English to French: Well, where to start describ...|Vous savez déjà que la grande graisse NADA passe comme un...|\n", - "|Translate English to French: I hoped for this show to be ...| J'espère que ce spectacle sera quelque peu réaliste|\n", - "| Translate English to French: All I have to say is one word| Je n'ai qu'à dire un mot|\n", - "|Translate English to French: Honestly awful film, bad edi...| l'instar de l'arrière-plan, il|\n", - "|Translate English to French: This critique tells the stor...|Cette critique raconte l'histoire de 4 petits amis qui on...|\n", - "|Translate English to French: This review contains a parti...| Cet examen contient un spoiler partiel|\n", - "|Translate English to French: I'm rather surprised that an...|Je suis plutôt surpris que quelqu'un ait trouvé ce film t...|\n", - "|Translate English to French: If you like bad movies (and ...|Si vous aimez des films mauvais (et vous devez regarder c...|\n", - "|Translate English to French: This is really bad, the char...| C'est vraiment mauvais, les personnages étaient bourds, l|\n", + "|Translate English to French: This is so overly clichéd yo...| Vous ne pouvez pas en tirer d'un tel cliché|\n", + "|Translate English to French: I am a big fan of The ABC Mo...| Je suis un grand fan du genre The ABC Movies of the Week|\n", + "|Translate English to French: In the early 1990's \"Step-by...| Au début des années 1990, «Step-by-Step» a été une|\n", + "|Translate English to French: When The Spirits Within was ...|Lorsque The Spirits Within a été publié, tout ce que vous...|\n", + "|Translate English to French: I like to think of myself as...| Je me considère comme un mauvais réalisateur de films|\n", + "|Translate English to French: This film did well at the bo...| Ce film a bien avancé à la salle de cinéma et|\n", + "|Translate English to French: Following the pleasingly atm...|Après l'original agréablement atmosphérique et la seconde...|\n", + "|Translate English to French: I like CKY and Viva La Bam, ...| Je m'aime CKY et Viva La Bam,|\n", + "|Translate English to French: I have read all of the revie...| J'ai lu tous les commentaires pour ce film direct à vidéo|\n", + "|Translate English to French: Yes, it was an awful movie, ...| Oui, c'était un film terrible, mais il y avait une chanson|\n", + "|Translate English to French: This was the most uninterest...| Ce fut le plus inquiétant et le plus inquiétant d'h|\n", + "|Translate English to French: I don't know if this excepti...|Je ne sais pas si ce film extrêmement tacheté était desti...|\n", + "|Translate English to French: Heart of Darkness Movie Revi...| Un livre connu pour son éloquence et ses concepts complexes|\n", + "| Translate English to French: A bad movie ABOUT a bad movie| Un mauvais film ABOUT a bad movie|\n", + "|Translate English to French: Apart from the fact that thi...|En plus du fait que ce film a été réalisé (je suppose quil s|\n", + "|Translate English to French: Watching this movie, you jus...|Vous devez simplement vous demander : « Que pense-t-il? » Il|\n", + "| Translate English to French: OK, lets start with the best| OK, s'il y a lieu de commencer par le meilleur|\n", + "|Translate English to French: Anna Christie (Greta Garbo) ...|Anna Christie (Greta Garbo) retourne pour voir son père C...|\n", + "| Translate English to French: C| C|\n", + "|Translate English to French: Tom and Jerry are transporti...| Tom et Jerry transportent des marchandises par avion en|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] } ], "source": [ @@ -1719,7 +1604,7 @@ }, { "cell_type": "code", - "execution_count": 56, + "execution_count": 48, "id": "425d3b28-7705-45ba-8a18-ad34fc895219", "metadata": {}, "outputs": [ @@ -1736,7 +1621,7 @@ "[True]" ] }, - "execution_count": 56, + "execution_count": 48, "metadata": {}, "output_type": "execute_result" } @@ -1760,7 +1645,7 @@ }, { "cell_type": "code", - "execution_count": 57, + "execution_count": 49, "id": "2dec80ca-7a7c-46a9-97c0-7afb1572f5b9", "metadata": {}, "outputs": [], @@ -1779,7 +1664,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "spark-dl-torch", "language": "python", "name": "python3" }, @@ -1793,7 +1678,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.10" + "version": "3.11.9" } }, "nbformat": 4, diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_tf/1/model.py b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_tf/1/model.py new file mode 100644 index 000000000..b788c8930 --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_tf/1/model.py @@ -0,0 +1,150 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import numpy as np +import json + +# triton_python_backend_utils is available in every Triton Python model. You +# need to use this module to create inference requests and responses. It also +# contains some utility functions for extracting information from model_config +# and converting Triton input/output types to numpy types. +import triton_python_backend_utils as pb_utils + + +class TritonPythonModel: + """Your Python model must use the same class name. Every Python model + that is created must have "TritonPythonModel" as the class name. + """ + + def initialize(self, args): + """`initialize` is called only once when the model is being loaded. + Implementing `initialize` function is optional. This function allows + the model to intialize any state associated with this model. + + Parameters + ---------- + args : dict + Both keys and values are strings. The dictionary keys and values are: + * model_config: A JSON string containing the model configuration + * model_instance_kind: A string containing model instance kind + * model_instance_device_id: A string containing model instance device ID + * model_repository: Model repository path + * model_version: Model version + * model_name: Model name + """ + import tensorflow as tf + # Enable GPU memory growth + gpus = tf.config.experimental.list_physical_devices('GPU') + if gpus: + try: + for gpu in gpus: + tf.config.experimental.set_memory_growth(gpu, True) + except RuntimeError as e: + print(e) + + print(tf.__version__) + + from transformers import AutoTokenizer, TFT5ForConditionalGeneration + + self.tokenizer = AutoTokenizer.from_pretrained("google-t5/t5-small") + self.model = TFT5ForConditionalGeneration.from_pretrained("google-t5/t5-small") + + # You must parse model_config. JSON string is not parsed here + self.model_config = model_config = json.loads(args['model_config']) + + # Get output configuration + output_config = pb_utils.get_output_config_by_name(model_config, "output") + + # Convert Triton types to numpy types + self.output_dtype = pb_utils.triton_string_to_numpy(output_config['data_type']) + + def execute(self, requests): + """`execute` MUST be implemented in every Python model. `execute` + function receives a list of pb_utils.InferenceRequest as the only + argument. This function is called when an inference request is made + for this model. Depending on the batching configuration (e.g. Dynamic + Batching) used, `requests` may contain multiple requests. Every + Python model, must create one pb_utils.InferenceResponse for every + pb_utils.InferenceRequest in `requests`. If there is an error, you can + set the error argument when creating a pb_utils.InferenceResponse + + Parameters + ---------- + requests : list + A list of pb_utils.InferenceRequest + + Returns + ------- + list + A list of pb_utils.InferenceResponse. The length of this list must + be the same as `requests` + """ + + output_dtype = self.output_dtype + + responses = [] + + # Every Python backend must iterate over everyone of the requests + # and create a pb_utils.InferenceResponse for each of them. + for request in requests: + # Get input numpy + sentence_input = pb_utils.get_input_tensor_by_name(request, "input") + sentences = list(sentence_input.as_numpy()) + sentences = np.squeeze(sentences, -1).tolist() + sentences = [s.decode('utf-8') for s in sentences] + + input_ids = self.tokenizer(sentences, + padding="longest", + max_length=512, + truncation=True, + return_tensors="tf").input_ids + output_ids = self.model.generate(input_ids, max_length=20) + outputs = np.array([self.tokenizer.decode(o, skip_special_tokens=True) for o in output_ids]) + + # Create output tensors. You need pb_utils.Tensor + # objects to create pb_utils.InferenceResponse. + output_tensor = pb_utils.Tensor("output", outputs.astype(output_dtype)) + + # Create InferenceResponse. You can set an error here in case + # there was a problem with handling this inference request. + # Below is an example of how you can set errors in inference + # response: + # + # pb_utils.InferenceResponse( + # output_tensors=..., TritonError("An error occured")) + inference_response = pb_utils.InferenceResponse(output_tensors=[output_tensor]) + responses.append(inference_response) + + # You should return a list of pb_utils.InferenceResponse. Length + # of this list must match the length of `requests` list. + return responses + + def finalize(self): + """`finalize` is called only once when the model is being unloaded. + Implementing `finalize` function is OPTIONAL. This function allows + the model to perform any necessary clean ups before exit. + """ + print('Cleaning up...') diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation/config.pbtxt b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_tf/config.pbtxt similarity index 98% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation/config.pbtxt rename to examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_tf/config.pbtxt index 380af49c5..88b87130f 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation/config.pbtxt +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_tf/config.pbtxt @@ -24,7 +24,7 @@ # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -name: "hf_generation" +name: "hf_generation_tf" backend: "python" max_batch_size: 8192 @@ -47,6 +47,6 @@ instance_group [{ kind: KIND_GPU }] parameters: { key: "EXECUTION_ENV_PATH", - value: {string_value: "$$TRITON_MODEL_DIRECTORY/../huggingface.tar.gz"} + value: {string_value: "$$TRITON_MODEL_DIRECTORY/../huggingface-tf.tar.gz"} } diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation/1/model.py b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_torch/1/model.py similarity index 98% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation/1/model.py rename to examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_torch/1/model.py index e7ae472fe..8e9604daa 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation/1/model.py +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_torch/1/model.py @@ -113,8 +113,9 @@ def execute(self, requests): input_ids = self.tokenizer(sentences, padding="longest", max_length=512, + truncation=True, return_tensors="pt").input_ids - output_ids = self.model.generate(input_ids) + output_ids = self.model.generate(input_ids, max_length=20) outputs = np.array([self.tokenizer.decode(o, skip_special_tokens=True) for o in output_ids]) # Create output tensors. You need pb_utils.Tensor diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_torch/config.pbtxt b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_torch/config.pbtxt new file mode 100644 index 000000000..47db54680 --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_generation_torch/config.pbtxt @@ -0,0 +1,52 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +name: "hf_generation_torch" +backend: "python" +max_batch_size: 8192 + +input [ + { + name: "input" + data_type: TYPE_STRING + dims: [1] + } +] +output [ + { + name: "output" + data_type: TYPE_STRING + dims: [1] + } +] + +instance_group [{ kind: KIND_GPU }] + +parameters: { + key: "EXECUTION_ENV_PATH", + value: {string_value: "$$TRITON_MODEL_DIRECTORY/../huggingface-torch.tar.gz"} +} + diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_tf/1/model.py b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_tf/1/model.py new file mode 100644 index 000000000..2a1bfda61 --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_tf/1/model.py @@ -0,0 +1,147 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +import numpy as np +import json + +# triton_python_backend_utils is available in every Triton Python model. You +# need to use this module to create inference requests and responses. It also +# contains some utility functions for extracting information from model_config +# and converting Triton input/output types to numpy types. +import triton_python_backend_utils as pb_utils + + +class TritonPythonModel: + """Your Python model must use the same class name. Every Python model + that is created must have "TritonPythonModel" as the class name. + """ + + def initialize(self, args): + """`initialize` is called only once when the model is being loaded. + Implementing `initialize` function is optional. This function allows + the model to intialize any state associated with this model. + + Parameters + ---------- + args : dict + Both keys and values are strings. The dictionary keys and values are: + * model_config: A JSON string containing the model configuration + * model_instance_kind: A string containing model instance kind + * model_instance_device_id: A string containing model instance device ID + * model_repository: Model repository path + * model_version: Model version + * model_name: Model name + """ + import tensorflow as tf + print("tf: {}".format(tf.__version__)) + + # Enable GPU memory growth + gpus = tf.config.experimental.list_physical_devices('GPU') + if gpus: + try: + for gpu in gpus: + tf.config.experimental.set_memory_growth(gpu, True) + except RuntimeError as e: + print(e) + + from transformers import pipeline + self.pipe = pipeline("sentiment-analysis", device=0) + + # You must parse model_config. JSON string is not parsed here + self.model_config = model_config = json.loads(args['model_config']) + + # Get output configuration + label_config = pb_utils.get_output_config_by_name(model_config, "label") + score_config = pb_utils.get_output_config_by_name(model_config, "score") + + # Convert Triton types to numpy types + self.label_dtype = pb_utils.triton_string_to_numpy(label_config['data_type']) + self.score_dtype = pb_utils.triton_string_to_numpy(score_config['data_type']) + + def execute(self, requests): + """`execute` MUST be implemented in every Python model. `execute` + function receives a list of pb_utils.InferenceRequest as the only + argument. This function is called when an inference request is made + for this model. Depending on the batching configuration (e.g. Dynamic + Batching) used, `requests` may contain multiple requests. Every + Python model, must create one pb_utils.InferenceResponse for every + pb_utils.InferenceRequest in `requests`. If there is an error, you can + set the error argument when creating a pb_utils.InferenceResponse + + Parameters + ---------- + requests : list + A list of pb_utils.InferenceRequest + + Returns + ------- + list + A list of pb_utils.InferenceResponse. The length of this list must + be the same as `requests` + """ + + label_dtype = self.label_dtype + score_dtype = self.score_dtype + + responses = [] + + # Every Python backend must iterate over everyone of the requests + # and create a pb_utils.InferenceResponse for each of them. + for request in requests: + # Get input numpy + sentence_input = pb_utils.get_input_tensor_by_name(request, "sentence") + sentences = [s.decode('utf-8') for s in sentence_input.as_numpy().flatten()] + + results = self.pipe(sentences) + + label = np.array([res['label'] for res in results]) + score = np.array([res['score'] for res in results]) + + # Create output tensors. You need pb_utils.Tensor + # objects to create pb_utils.InferenceResponse. + label_tensor = pb_utils.Tensor("label", label.astype(label_dtype)) + score_tensor = pb_utils.Tensor("score", score.astype(score_dtype)) + + # Create InferenceResponse. You can set an error here in case + # there was a problem with handling this inference request. + # Below is an example of how you can set errors in inference + # response: + # + # pb_utils.InferenceResponse( + # output_tensors=..., TritonError("An error occured")) + inference_response = pb_utils.InferenceResponse(output_tensors=[label_tensor, score_tensor]) + responses.append(inference_response) + + # You should return a list of pb_utils.InferenceResponse. Length + # of this list must match the length of `requests` list. + return responses + + def finalize(self): + """`finalize` is called only once when the model is being unloaded. + Implementing `finalize` function is OPTIONAL. This function allows + the model to perform any necessary clean ups before exit. + """ + print('Cleaning up...') diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline/config.pbtxt b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_tf/config.pbtxt similarity index 98% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline/config.pbtxt rename to examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_tf/config.pbtxt index 5caae44f2..df7082ca4 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline/config.pbtxt +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_tf/config.pbtxt @@ -24,7 +24,7 @@ # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -name: "hf_pipeline" +name: "hf_pipeline_tf" backend: "python" max_batch_size: 8192 @@ -52,6 +52,6 @@ instance_group [{ kind: KIND_GPU }] parameters: { key: "EXECUTION_ENV_PATH", - value: {string_value: "$$TRITON_MODEL_DIRECTORY/../huggingface.tar.gz"} + value: {string_value: "$$TRITON_MODEL_DIRECTORY/../huggingface-tf.tar.gz"} } diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline/1/model.py b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_torch/1/model.py similarity index 96% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline/1/model.py rename to examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_torch/1/model.py index 79faa0306..f01886c91 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline/1/model.py +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_torch/1/model.py @@ -63,7 +63,7 @@ def initialize(self, args): print("transformers: {}".format(transformers.__version__)) from transformers import pipeline - self.pipe = pipeline("text-classification", device=0) + self.pipe = pipeline("sentiment-analysis", device=0) # You must parse model_config. JSON string is not parsed here self.model_config = model_config = json.loads(args['model_config']) @@ -108,9 +108,7 @@ def execute(self, requests): for request in requests: # Get input numpy sentence_input = pb_utils.get_input_tensor_by_name(request, "sentence") - sentences = list(sentence_input.as_numpy()) - sentences = np.squeeze(sentences).tolist() - sentences = [s.decode('utf-8') for s in sentences] + sentences = [s.decode('utf-8') for s in sentence_input.as_numpy().flatten()] results = self.pipe(sentences) diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_torch/config.pbtxt b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_torch/config.pbtxt new file mode 100644 index 000000000..4e54607d2 --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_pipeline_torch/config.pbtxt @@ -0,0 +1,57 @@ +# Copyright (c) 2023, NVIDIA CORPORATION. All rights reserved. +# +# Redistribution and use in source and binary forms, with or without +# modification, are permitted provided that the following conditions +# are met: +# * Redistributions of source code must retain the above copyright +# notice, this list of conditions and the following disclaimer. +# * Redistributions in binary form must reproduce the above copyright +# notice, this list of conditions and the following disclaimer in the +# documentation and/or other materials provided with the distribution. +# * Neither the name of NVIDIA CORPORATION nor the names of its +# contributors may be used to endorse or promote products derived +# from this software without specific prior written permission. +# +# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS ``AS IS'' AND ANY +# EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR +# PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR +# CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, +# EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, +# PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR +# PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY +# OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. + +name: "hf_pipeline_torch" +backend: "python" +max_batch_size: 8192 + +input [ + { + name: "sentence" + data_type: TYPE_STRING + dims: [1] + } +] +output [ + { + name: "label" + data_type: TYPE_STRING + dims: [1] + }, + { + name: "score" + data_type: TYPE_FP32 + dims: [1] + } +] + +instance_group [{ kind: KIND_GPU }] + +parameters: { + key: "EXECUTION_ENV_PATH", + value: {string_value: "$$TRITON_MODEL_DIRECTORY/../huggingface-torch.tar.gz"} +} + diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer/1/model.py b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer_torch/1/model.py similarity index 100% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer/1/model.py rename to examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer_torch/1/model.py diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer/config.pbtxt b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer_torch/config.pbtxt similarity index 97% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer/config.pbtxt rename to examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer_torch/config.pbtxt index ccdee3a50..798cf4fc7 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer/config.pbtxt +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/models_config/hf_transformer_torch/config.pbtxt @@ -24,7 +24,7 @@ # (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE # OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. -name: "hf_transformer" +name: "hf_transformer_torch" backend: "python" max_batch_size: 8192 @@ -47,6 +47,6 @@ instance_group [{ kind: KIND_GPU }] parameters: { key: "EXECUTION_ENV_PATH", - value: {string_value: "$$TRITON_MODEL_DIRECTORY/../huggingface.tar.gz"} + value: {string_value: "$$TRITON_MODEL_DIRECTORY/../huggingface-torch.tar.gz"} } diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/pipelines_tf.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/pipelines_tf.ipynb new file mode 100644 index 000000000..115cfffc4 --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/pipelines_tf.ipynb @@ -0,0 +1,1056 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "60f7ac5d-4a95-4170-a0ac-a7faac9d9ef4", + "metadata": {}, + "source": [ + "# PySpark Huggingface Inferencing\n", + "### Text Classification using Pipelines with Tensorflow\n", + "\n", + "Based on: https://huggingface.co/docs/transformers/quicktour#pipeline-usage" + ] + }, + { + "cell_type": "markdown", + "id": "1799fd4f", + "metadata": {}, + "source": [ + "### Using TensorFlow\n", + "Note that cuFFT/cuDNN/cuBLAS registration errors are expected with `tf=2.17.0` and will not affect behavior, as noted in [this issue.](https://github.com/tensorflow/tensorflow/issues/62075) \n", + "This notebook does not demonstrate inference with TensorRT, as [TF-TRT](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#tensorrt-10) does not yet support `tf=2.17.0`. See the `pytorch` notebooks for TensorRT demos." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "0dd0f77b-ee1b-4477-a038-d25a4f1da0ea", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 16:47:48.209366: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", + "2024-10-03 16:47:48.215921: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", + "2024-10-03 16:47:48.223519: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", + "2024-10-03 16:47:48.225906: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n", + "2024-10-03 16:47:48.231640: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", + "To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", + "2024-10-03 16:47:48.625790: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" + ] + } + ], + "source": [ + "import tensorflow as tf\n", + "from transformers import pipeline" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "d80fc3f8", + "metadata": {}, + "outputs": [], + "source": [ + "# set device if tensorflow gpu is available\n", + "device = 0 if tf.config.list_physical_devices('GPU') else -1" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "e60a2877", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2.17.0\n" + ] + } + ], + "source": [ + "print(tf.__version__)\n", + "\n", + "# Enable GPU memory growth\n", + "gpus = tf.config.experimental.list_physical_devices('GPU')\n", + "if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "553b28d2-a5d1-4d07-8a49-8f82b808e738", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "No model was supplied, defaulted to distilbert/distilbert-base-uncased-finetuned-sst-2-english and revision 714eb0f (https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english).\n", + "Using a pipeline without specifying a model name and revision in production is not recommended.\n", + "2024-10-03 16:47:49.863791: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2021] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 46447 MB memory: -> device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:01:00.0, compute capability: 8.6\n", + "All PyTorch model weights were used when initializing TFDistilBertForSequenceClassification.\n", + "\n", + "All the weights of TFDistilBertForSequenceClassification were initialized from the PyTorch model.\n", + "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFDistilBertForSequenceClassification for predictions without further training.\n" + ] + } + ], + "source": [ + "classifier = pipeline(\"sentiment-analysis\", device=device)" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "3b91fe91-b725-4564-ae93-56e3fb51e47c", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[{'label': 'POSITIVE', 'score': 0.9997794032096863}]" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "classifier((\"We are very happy to show you the 🤗 Transformers library.\"))" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "0be39eb3-462c-42ff-b8f4-09f4e4fe3a3c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "label: POSITIVE, with score: 0.9998\n", + "label: NEGATIVE, with score: 0.5282\n" + ] + } + ], + "source": [ + "results = classifier([\"We are very happy to show you the 🤗 Transformers library.\", \"We hope you don't hate it.\"])\n", + "for result in results:\n", + " print(f\"label: {result['label']}, with score: {round(result['score'], 4)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "30c90100", + "metadata": {}, + "source": [ + "#### Use another model and tokenizer in the pipeline" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "cd9d3349", + "metadata": {}, + "outputs": [], + "source": [ + "model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "99e21b58", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "Some layers from the model checkpoint at nlptown/bert-base-multilingual-uncased-sentiment were not used when initializing TFBertForSequenceClassification: ['dropout_37']\n", + "- This IS expected if you are initializing TFBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n", + "- This IS NOT expected if you are initializing TFBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n", + "All the layers of TFBertForSequenceClassification were initialized from the model checkpoint at nlptown/bert-base-multilingual-uncased-sentiment.\n", + "If your task is similar to the task the model of the checkpoint was trained on, you can already use TFBertForSequenceClassification for predictions without further training.\n" + ] + } + ], + "source": [ + "from transformers import AutoTokenizer, TFAutoModelForSequenceClassification\n", + "\n", + "model = TFAutoModelForSequenceClassification.from_pretrained(model_name)\n", + "tokenizer = AutoTokenizer.from_pretrained(model_name)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "31079133", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "[{'label': '5 stars', 'score': 0.7272655963897705}]" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "classifier = pipeline(\"sentiment-analysis\", model=model, tokenizer=tokenizer)\n", + "classifier(\"Nous sommes très heureux de vous présenter la bibliothèque 🤗 Transformers.\")" + ] + }, + { + "cell_type": "markdown", + "id": "ae92b15e-0da0-46c3-81a3-fabaedbfc42c", + "metadata": {}, + "source": [ + "## Inference using Spark DL API" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "69dd6a1a-f450-47f0-9dbf-ad250585a011", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import pandas as pd\n", + "from pyspark.sql.functions import col, struct, pandas_udf\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.types import FloatType, StringType, StructField, StructType\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6e0e0dd7", + "metadata": {}, + "outputs": [], + "source": [ + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "42d70208", + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "# Load the IMDB dataset\n", + "data = load_dataset(\"imdb\", split=\"test\")\n", + "\n", + "lines = []\n", + "for example in data:\n", + " # first sentence only\n", + " lines.append([example[\"text\"]])\n", + "\n", + "len(lines)\n", + "\n", + "df = spark.createDataFrame(lines, ['lines']).repartition(8).cache()" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "ac24f3c2", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "24/10/03 16:47:58 WARN TaskSetManager: Stage 0 contains a task of very large size (3860 KiB). The maximum recommended task size is 1000 KiB.\n", + " \r" + ] + } + ], + "source": [ + "df.write.mode(\"overwrite\").parquet(\"imdb_test\")" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "9665b7b6-d7e9-4bd4-b29d-7a449ac5b574", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+--------------------------------------------------------------------------------+\n", + "| sentence|\n", + "+--------------------------------------------------------------------------------+\n", + "| |\n", + "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pi...|\n", + "|I watched this movie to see the direction one of the most promising young tal...|\n", + "| This movie makes you wish imdb would let you vote a zero|\n", + "|I never want to see this movie again!

Not only is it dreadfully ba...|\n", + "|(As a note, I'd like to say that I saw this movie at my annual church camp, w...|\n", + "| Don't get me wrong, I love the TV series of League Of Gentlemen|\n", + "|Did you ever think, like after watching a horror movie with a group of friend...|\n", + "| Awful, awful, awful|\n", + "|This movie seems a little clunky around the edges, like not quite enough zani...|\n", + "|I rented this movie hoping that it would provide some good entertainment and ...|\n", + "|Well, where to start describing this celluloid debacle? You already know the ...|\n", + "| I hoped for this show to be somewhat realistic|\n", + "| All I have to say is one word|\n", + "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy scr...|\n", + "|This critique tells the story of 4 little friends who went to watch Angels an...|\n", + "| This review contains a partial spoiler|\n", + "| I'm rather surprised that anybody found this film touching or moving|\n", + "| If you like bad movies (and you must to watch this one) here's a good one|\n", + "|This is really bad, the characters were bland, the story was boring, and ther...|\n", + "+--------------------------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + } + ], + "source": [ + "# only use first sentence of IMDB reviews\n", + "@pandas_udf(\"string\")\n", + "def first_sentence(text: pd.Series) -> pd.Series:\n", + " return pd.Series([s.split(\".\")[0] for s in text])\n", + "\n", + "df = spark.read.parquet(\"imdb_test\").withColumn(\"sentence\", first_sentence(col(\"lines\"))).select(\"sentence\").limit(100).cache()\n", + "df.show(truncate=80)" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "0da9d25c-5ebe-4503-bb19-154fcc047cbf", + "metadata": {}, + "outputs": [], + "source": [ + "def predict_batch_fn():\n", + " import tensorflow as tf\n", + " from transformers import pipeline\n", + "\n", + " # Enable GPU memory growth\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + " \n", + " device = 0 if tf.config.list_physical_devices('GPU') else -1\n", + " pipe = pipeline(\"sentiment-analysis\", device=device)\n", + " def predict(inputs):\n", + " return pipe(inputs.tolist())\n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "78afef29-ee30-4267-9fb6-be2dcb86cbba", + "metadata": {}, + "outputs": [], + "source": [ + "classify = predict_batch_udf(predict_batch_fn,\n", + " return_type=StructType([\n", + " StructField(\"label\", StringType(), True),\n", + " StructField(\"score\", FloatType(), True)\n", + " ]),\n", + " batch_size=10)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "a5bc327e-89cf-4731-82e6-e66cb93deef1", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 11:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 9.15 ms, sys: 6.76 ms, total: 15.9 ms\n", + "Wall time: 5 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# note: expanding the \"struct\" return_type to top-level columns\n", + "preds = df.withColumn(\"preds\", classify(struct(\"sentence\"))).select(\"sentence\", \"preds.*\")\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "ac642895-cfd6-47ee-9b21-02e7835424e4", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 13:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 4.86 ms, sys: 2.19 ms, total: 7.05 ms\n", + "Wall time: 2.81 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# note: expanding the \"struct\" return_type to top-level columns\n", + "preds = df.withColumn(\"preds\", classify(\"sentence\")).select(\"sentence\", \"preds.*\")\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "76a44d80-d5db-405f-989c-7246379cfb95", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 15:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 3.91 ms, sys: 1.96 ms, total: 5.87 ms\n", + "Wall time: 2.76 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# note: expanding the \"struct\" return_type to top-level columns\n", + "preds = df.withColumn(\"preds\", classify(col(\"sentence\"))).select(\"sentence\", \"preds.*\")\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "c01761b3-c766-46b0-ae0b-fcf968ffb3a1", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+--------------------------------------------------------------------------------+--------+----------+\n", + "| sentence| label| score|\n", + "+--------------------------------------------------------------------------------+--------+----------+\n", + "| |POSITIVE|0.74807304|\n", + "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pi...|NEGATIVE| 0.9996724|\n", + "|I watched this movie to see the direction one of the most promising young tal...|POSITIVE| 0.9994948|\n", + "| This movie makes you wish imdb would let you vote a zero|NEGATIVE| 0.9981299|\n", + "|I never want to see this movie again!

Not only is it dreadfully ba...|NEGATIVE|0.99883264|\n", + "|(As a note, I'd like to say that I saw this movie at my annual church camp, w...|POSITIVE| 0.9901753|\n", + "| Don't get me wrong, I love the TV series of League Of Gentlemen|POSITIVE|0.99983096|\n", + "|Did you ever think, like after watching a horror movie with a group of friend...|POSITIVE| 0.9992768|\n", + "| Awful, awful, awful|NEGATIVE| 0.9997433|\n", + "|This movie seems a little clunky around the edges, like not quite enough zani...|NEGATIVE| 0.9996525|\n", + "|I rented this movie hoping that it would provide some good entertainment and ...|NEGATIVE|0.99643254|\n", + "|Well, where to start describing this celluloid debacle? You already know the ...|NEGATIVE|0.99973005|\n", + "| I hoped for this show to be somewhat realistic|POSITIVE| 0.8417903|\n", + "| All I have to say is one word|NEGATIVE|0.97844803|\n", + "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy scr...|NEGATIVE| 0.9997701|\n", + "|This critique tells the story of 4 little friends who went to watch Angels an...|POSITIVE| 0.9942386|\n", + "| This review contains a partial spoiler|NEGATIVE|0.99620205|\n", + "| I'm rather surprised that anybody found this film touching or moving|POSITIVE|0.83874947|\n", + "| If you like bad movies (and you must to watch this one) here's a good one|POSITIVE| 0.9936475|\n", + "|This is really bad, the characters were bland, the story was boring, and ther...|NEGATIVE|0.99953806|\n", + "+--------------------------------------------------------------------------------+--------+----------+\n", + "only showing top 20 rows\n", + "\n" + ] + } + ], + "source": [ + "preds.show(truncate=80)" + ] + }, + { + "cell_type": "markdown", + "id": "eb826fde-99d9-43fe-8ddc-f5acbe76b4e9", + "metadata": {}, + "source": [ + "### Using Triton Inference Server\n", + "\n", + "Note: you can restart the kernel and run from this point to simulate running in a different node or environment. " + ] + }, + { + "cell_type": "markdown", + "id": "10368010-f94d-4167-91a1-2cf9ed91a2c9", + "metadata": {}, + "source": [ + "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments) with the compatible versions of Python/Numpy for Triton 24.08, using a conda-pack environment created as follows:\n", + "```\n", + "conda create -n huggingface-tf -c conda-forge python=3.10.0\n", + "conda activate huggingface-tf\n", + "\n", + "export PYTHONNOUSERSITE=True\n", + "pip install numpy==1.26.4 tensorflow[and-cuda] tf-keras transformers conda-pack\n", + "\n", + "conda-pack # huggingface-tf.tar.gz\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "4d4be844-4b8c-47df-bd09-0c280c7ff16b", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import pandas as pd\n", + "import os\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import col, struct, pandas_udf\n", + "from pyspark.sql.types import FloatType, StringType, StructField, StructType" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "7e53df9f-43cb-4c38-b8ac-dc2cbad99815", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "# copy custom model to expected layout for Triton\n", + "rm -rf models\n", + "mkdir -p models\n", + "cp -r models_config/hf_pipeline_tf models\n", + "\n", + "# add custom execution environment\n", + "cp huggingface-tf.tar.gz models" + ] + }, + { + "cell_type": "markdown", + "id": "db4a5b06-126a-4bc4-baae-a45ea30832a7", + "metadata": { + "tags": [] + }, + "source": [ + "#### Start Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "144acb8e-4c08-40fc-a9ed-f721c409ee68", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "[True]" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "num_executors = 1\n", + "triton_models_dir = \"{}/models\".format(os.getcwd())\n", + "huggingface_cache_dir = \"{}/.cache/huggingface\".format(os.path.expanduser('~'))\n", + "nodeRDD = sc.parallelize(list(range(num_executors)), num_executors)\n", + "\n", + "def start_triton(it):\n", + " import docker\n", + " import time\n", + " import tritonclient.grpc as grpcclient\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " if containers:\n", + " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", + " else:\n", + " container=client.containers.run(\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", + " detach=True,\n", + " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", + " environment=[\n", + " \"TRANSFORMERS_CACHE=/cache\"\n", + " ],\n", + " name=\"spark-triton\",\n", + " network_mode=\"host\",\n", + " remove=True,\n", + " shm_size=\"256M\",\n", + " volumes={\n", + " triton_models_dir: {\"bind\": \"/models\", \"mode\": \"ro\"},\n", + " huggingface_cache_dir: {\"bind\": \"/cache\", \"mode\": \"rw\"}\n", + " }\n", + " )\n", + " print(\">>>> starting triton: {}\".format(container.short_id))\n", + " # wait for triton to be running\n", + " time.sleep(15)\n", + " \n", + " client = grpcclient.InferenceServerClient(\"localhost:8001\")\n", + " \n", + " elapsed = 0\n", + " timeout = 120\n", + " ready = False\n", + " while not ready and elapsed < timeout:\n", + " try:\n", + " time.sleep(5)\n", + " elapsed += 5\n", + " ready = client.is_server_ready()\n", + " except Exception as e:\n", + " pass\n", + "\n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(start_triton).collect()" + ] + }, + { + "cell_type": "markdown", + "id": "c24d77ab-60d3-45eb-a9c2-dc811eca0af4", + "metadata": {}, + "source": [ + "#### Run inference" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "d53fb283-bf9e-4571-8c68-b75a41f1f067", + "metadata": {}, + "outputs": [], + "source": [ + "# only use first sentence of IMDB reviews\n", + "@pandas_udf(\"string\")\n", + "def first_sentence(text: pd.Series) -> pd.Series:\n", + " return pd.Series([s.split(\".\")[0] for s in text])\n", + "\n", + "df = spark.read.parquet(\"imdb_test\").withColumn(\"sentence\", first_sentence(col(\"lines\"))).select(\"sentence\").limit(1000)" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "29b0cc0d-c480-4e4a-bd41-207dc314cba5", + "metadata": {}, + "outputs": [], + "source": [ + "def triton_fn(triton_uri, model_name):\n", + " import numpy as np\n", + " import tritonclient.grpc as grpcclient\n", + " \n", + " np_types = {\n", + " \"BOOL\": np.dtype(np.bool_),\n", + " \"INT8\": np.dtype(np.int8),\n", + " \"INT16\": np.dtype(np.int16),\n", + " \"INT32\": np.dtype(np.int32),\n", + " \"INT64\": np.dtype(np.int64),\n", + " \"FP16\": np.dtype(np.float16),\n", + " \"FP32\": np.dtype(np.float32),\n", + " \"FP64\": np.dtype(np.float64),\n", + " \"FP64\": np.dtype(np.double),\n", + " \"BYTES\": np.dtype(object)\n", + " }\n", + "\n", + " client = grpcclient.InferenceServerClient(triton_uri)\n", + " model_meta = client.get_model_metadata(model_name)\n", + " \n", + " def predict(inputs):\n", + " if isinstance(inputs, np.ndarray):\n", + " # single ndarray input\n", + " request = [grpcclient.InferInput(model_meta.inputs[0].name, inputs.shape, model_meta.inputs[0].datatype)]\n", + " request[0].set_data_from_numpy(inputs.astype(np_types[model_meta.inputs[0].datatype]))\n", + " else:\n", + " # dict of multiple ndarray inputs\n", + " request = [grpcclient.InferInput(i.name, inputs[i.name].shape, i.datatype) for i in model_meta.inputs]\n", + " for i in request:\n", + " i.set_data_from_numpy(inputs[i.name()].astype(np_types[i.datatype()]))\n", + " \n", + " response = client.infer(model_name, inputs=request)\n", + " \n", + " if len(model_meta.outputs) > 1:\n", + " # return dictionary of numpy arrays\n", + " return {o.name: response.as_numpy(o.name) for o in model_meta.outputs}\n", + " else:\n", + " # return single numpy array\n", + " return response.as_numpy(model_meta.outputs[0].name)\n", + " \n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "3930cfcd-3284-4c6a-a9b5-36b8053fe899", + "metadata": {}, + "outputs": [], + "source": [ + "from functools import partial\n", + "\n", + "classify = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"hf_pipeline_tf\"),\n", + " return_type=StructType([\n", + " StructField(\"label\", StringType(), True),\n", + " StructField(\"score\", FloatType(), True)\n", + " ]),\n", + " input_tensor_shapes=[[1]],\n", + " batch_size=100)" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "8eecbf23-4e9e-4d4c-8645-98209b25db2c", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 20:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 22.5 ms, sys: 5.9 ms, total: 28.4 ms\n", + "Wall time: 24.6 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# first pass caches model/fn\n", + "# note: expanding the \"struct\" return_type to top-level columns\n", + "preds = df.withColumn(\"preds\", classify(struct(\"sentence\"))).select(\"sentence\", \"preds.*\")\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "566ba28c-0ca4-4479-a24a-c8a362228b89", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 21:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 12.2 ms, sys: 10.1 ms, total: 22.3 ms\n", + "Wall time: 23.8 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# note: expanding the \"struct\" return_type to top-level columns\n", + "preds = df.withColumn(\"preds\", classify(\"sentence\")).select(\"sentence\", \"preds.*\")\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "44c7e776-08da-484a-ba07-9d6add1a0f15", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 22:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 8.74 ms, sys: 8.23 ms, total: 17 ms\n", + "Wall time: 23.8 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# note: expanding the \"struct\" return_type to top-level columns\n", + "preds = df.withColumn(\"preds\", classify(col(\"sentence\"))).select(\"sentence\", \"preds.*\")\n", + "results = preds.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "f61d79f8-661e-4d9e-a3aa-c0754b854603", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 23:> (0 + 1) / 1]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+----------+\n", + "|sentence |label |score |\n", + "+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+----------+\n", + "| |POSITIVE|0.74807304|\n", + "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story without the gratuitous sex scenes, either hard core or soft core, therefore reads like a public information film from the fifties, give this a wide miss, use a barge pole if you can|NEGATIVE|0.9996724 |\n", + "|I watched this movie to see the direction one of the most promising young talents in movies was going |POSITIVE|0.9994948 |\n", + "|This movie makes you wish imdb would let you vote a zero |NEGATIVE|0.9981299 |\n", + "|I never want to see this movie again!

Not only is it dreadfully bad, but I can't stand seeing my hero Stan Laurel looking so old and sick |NEGATIVE|0.99883264|\n", + "|(As a note, I'd like to say that I saw this movie at my annual church camp, where the entire youth group laughed at it |POSITIVE|0.9901753 |\n", + "|Don't get me wrong, I love the TV series of League Of Gentlemen |POSITIVE|0.99983096|\n", + "|Did you ever think, like after watching a horror movie with a group of friends: \"Wow, this is so cool! We have got to make a splatter horror movie ourselves some day soon |POSITIVE|0.9992768 |\n", + "|Awful, awful, awful |NEGATIVE|0.9997433 |\n", + "|This movie seems a little clunky around the edges, like not quite enough zaniness was thrown it when it should have been |NEGATIVE|0.9996525 |\n", + "|I rented this movie hoping that it would provide some good entertainment and some cool poker knowledge or stories |NEGATIVE|0.99643254|\n", + "|Well, where to start describing this celluloid debacle? You already know the big fat NADA passing as a plot, so let's jut point out that this is so PC it's offensive |NEGATIVE|0.99973005|\n", + "|I hoped for this show to be somewhat realistic |POSITIVE|0.8417903 |\n", + "|All I have to say is one word |NEGATIVE|0.97844803|\n", + "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay |NEGATIVE|0.9997701 |\n", + "|This critique tells the story of 4 little friends who went to watch Angels and Demons the movie on the first night it came out, even though it was a school night, because \"Angels and Demons is worth it |POSITIVE|0.9942386 |\n", + "|This review contains a partial spoiler |NEGATIVE|0.99620205|\n", + "|I'm rather surprised that anybody found this film touching or moving |POSITIVE|0.83874947|\n", + "|If you like bad movies (and you must to watch this one) here's a good one |POSITIVE|0.9936475 |\n", + "|This is really bad, the characters were bland, the story was boring, and there is no sex scene |NEGATIVE|0.99953806|\n", + "+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+----------+\n", + "only showing top 20 rows\n", + "\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "preds.show(truncate=False)" + ] + }, + { + "cell_type": "markdown", + "id": "e197c146-1794-47f0-bcd9-7e8d8ab8625f", + "metadata": { + "tags": [] + }, + "source": [ + "#### Stop Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "425d3b28-7705-45ba-8a18-ad34fc895219", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "[True]" + ] + }, + "execution_count": 32, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def stop_triton(it):\n", + " import docker\n", + " import time\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " print(\">>>> stopping containers: {}\".format([c.short_id for c in containers]))\n", + " if containers:\n", + " container=containers[0]\n", + " container.stop(timeout=120)\n", + "\n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(stop_triton).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "9f19643c-4ee4-44f2-b762-2078c0c8eba9", + "metadata": {}, + "outputs": [], + "source": [ + "spark.stop()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6a538c47-317d-4cac-b9b9-559e88677518", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "spark-dl-tf", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/pipelines.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/pipelines_torch.ipynb similarity index 65% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/pipelines.ipynb rename to examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/pipelines_torch.ipynb index 51b8feb8e..47f45b670 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/pipelines.ipynb +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/pipelines_torch.ipynb @@ -6,37 +6,36 @@ "metadata": {}, "source": [ "# PySpark Huggingface Inferencing\n", - "### Text Classification using Pipelines\n", + "### Text Classification using Pipelines with PyTorch\n", "\n", "Based on: https://huggingface.co/docs/transformers/quicktour#pipeline-usage" ] }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "id": "0dd0f77b-ee1b-4477-a038-d25a4f1da0ea", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/home/leey/.pyenv/versions/3.9.10/envs/spark_rapids_examples/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", - " from .autonotebook import tqdm as notebook_tqdm\n" - ] - } - ], + "outputs": [], "source": [ - "import pandas as pd\n", - "\n", - "from inspect import signature\n", - "from pyspark.sql.functions import col, pandas_udf\n", + "import torch\n", "from transformers import pipeline" ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, + "id": "e1f756c6", + "metadata": {}, + "outputs": [], + "source": [ + "# set device if gpu is available\n", + "device = 0 if torch.cuda.is_available() else -1" + ] + }, + { + "cell_type": "code", + "execution_count": 4, "id": "553b28d2-a5d1-4d07-8a49-8f82b808e738", "metadata": {}, "outputs": [ @@ -44,57 +43,108 @@ "name": "stderr", "output_type": "stream", "text": [ - "No model was supplied, defaulted to distilbert-base-uncased-finetuned-sst-2-english and revision af0f99b (https://huggingface.co/distilbert-base-uncased-finetuned-sst-2-english).\n", - "Using a pipeline without specifying a model name and revision in production is not recommended.\n", - "Xformers is not installed correctly. If you want to use memorry_efficient_attention to accelerate training use the following command to install Xformers\n", - "pip install xformers.\n" + "No model was supplied, defaulted to distilbert/distilbert-base-uncased-finetuned-sst-2-english and revision 714eb0f (https://huggingface.co/distilbert/distilbert-base-uncased-finetuned-sst-2-english).\n", + "Using a pipeline without specifying a model name and revision in production is not recommended.\n" ] } ], "source": [ - "pipe = pipeline(\"text-classification\")" + "classifier = pipeline(\"sentiment-analysis\", device=device)" ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 5, "id": "3b91fe91-b725-4564-ae93-56e3fb51e47c", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "[{'label': 'POSITIVE', 'score': 0.9994712471961975}]" + "[{'label': 'POSITIVE', 'score': 0.9997795224189758}]" ] }, - "execution_count": 3, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "pipe(\"What can I say that hasn't been said already. I think this place is totally worth the hype.\")" + "classifier((\"We are very happy to show you the 🤗 Transformers library.\"))" ] }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 6, "id": "0be39eb3-462c-42ff-b8f4-09f4e4fe3a3c", "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "label: POSITIVE, with score: 0.9998\n", + "label: NEGATIVE, with score: 0.5309\n" + ] + } + ], + "source": [ + "results = classifier([\"We are very happy to show you the 🤗 Transformers library.\", \"We hope you don't hate it.\"])\n", + "for result in results:\n", + " print(f\"label: {result['label']}, with score: {round(result['score'], 4)}\")" + ] + }, + { + "cell_type": "markdown", + "id": "f752f929", + "metadata": {}, + "source": [ + "#### Use another model and tokenizer in the pipeline" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "9861865f", + "metadata": {}, + "outputs": [], + "source": [ + "model_name = \"nlptown/bert-base-multilingual-uncased-sentiment\"" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "506e7834", + "metadata": {}, + "outputs": [], + "source": [ + "from transformers import AutoTokenizer, AutoModelForSequenceClassification\n", + "\n", + "model = AutoModelForSequenceClassification.from_pretrained(model_name)\n", + "tokenizer = AutoTokenizer.from_pretrained(model_name)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "312017fc", + "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "[{'label': 'NEGATIVE', 'score': 0.9997401833534241}]" + "[{'label': '5 stars', 'score': 0.7272652983665466}]" ] }, - "execution_count": 4, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "pipe(\"I will not say much about this film, because there is not much to say, because there is not much there to talk about.\")" + "classifier = pipeline(\"sentiment-analysis\", model=model, tokenizer=tokenizer, device=device)\n", + "classifier(\"Nous sommes très heureux de vous présenter la bibliothèque 🤗 Transformers.\")" ] }, { @@ -107,7 +157,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 10, "id": "69dd6a1a-f450-47f0-9dbf-ad250585a011", "metadata": {}, "outputs": [], @@ -115,12 +165,85 @@ "import pandas as pd\n", "from pyspark.sql.functions import col, struct, pandas_udf\n", "from pyspark.ml.functions import predict_batch_udf\n", - "from pyspark.sql.types import FloatType, StringType, StructField, StructType" + "from pyspark.sql.types import FloatType, StringType, StructField, StructType\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf\n", + "import os" ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": null, + "id": "6e0e0dd7", + "metadata": {}, + "outputs": [], + "source": [ + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "42d70208", + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "# Load the IMDB dataset\n", + "data = load_dataset(\"imdb\", split=\"test\")\n", + "\n", + "lines = []\n", + "for example in data:\n", + " # first sentence only\n", + " lines.append([example[\"text\"]])\n", + "\n", + "len(lines)\n", + "\n", + "df = spark.createDataFrame(lines, ['lines']).repartition(8).cache()" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "ac24f3c2", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "24/10/03 16:44:02 WARN TaskSetManager: Stage 0 contains a task of very large size (3860 KiB). The maximum recommended task size is 1000 KiB.\n", + " \r" + ] + } + ], + "source": [ + "df.write.mode(\"overwrite\").parquet(\"imdb_test\")" + ] + }, + { + "cell_type": "code", + "execution_count": 14, "id": "9665b7b6-d7e9-4bd4-b29d-7a449ac5b574", "metadata": {}, "outputs": [ @@ -128,7 +251,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 1:====================================================> (9 + 1) / 10]\r" + " \r" ] }, { @@ -162,13 +285,6 @@ "only showing top 20 rows\n", "\n" ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] } ], "source": [ @@ -183,14 +299,17 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 15, "id": "0da9d25c-5ebe-4503-bb19-154fcc047cbf", "metadata": {}, "outputs": [], "source": [ "def predict_batch_fn():\n", + " import torch\n", " from transformers import pipeline\n", - " pipe = pipeline(\"text-classification\")\n", + " \n", + " device = 0 if torch.cuda.is_available() else -1\n", + " pipe = pipeline(\"sentiment-analysis\", device=device)\n", " def predict(inputs):\n", " return pipe(inputs.tolist())\n", " return predict" @@ -198,7 +317,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 16, "id": "78afef29-ee30-4267-9fb6-be2dcb86cbba", "metadata": {}, "outputs": [], @@ -213,7 +332,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 17, "id": "a5bc327e-89cf-4731-82e6-e66cb93deef1", "metadata": {}, "outputs": [ @@ -221,15 +340,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 4:> (0 + 1) / 1]\r" + "[Stage 11:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 21.2 ms, sys: 4.33 ms, total: 25.5 ms\n", - "Wall time: 12.3 s\n" + "CPU times: user 12.6 ms, sys: 2.39 ms, total: 15 ms\n", + "Wall time: 2.02 s\n" ] }, { @@ -249,30 +368,16 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 18, "id": "ac642895-cfd6-47ee-9b21-02e7835424e4", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[Stage 6:> (0 + 1) / 1]\r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 8.67 ms, sys: 4.24 ms, total: 12.9 ms\n", - "Wall time: 5.44 s\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" + "CPU times: user 2.13 ms, sys: 1.06 ms, total: 3.19 ms\n", + "Wall time: 237 ms\n" ] } ], @@ -285,30 +390,16 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 19, "id": "76a44d80-d5db-405f-989c-7246379cfb95", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[Stage 8:> (0 + 1) / 1]\r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 13.8 ms, sys: 1.52 ms, total: 15.4 ms\n", - "Wall time: 5.46 s\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" + "CPU times: user 2.28 ms, sys: 790 μs, total: 3.07 ms\n", + "Wall time: 230 ms\n" ] } ], @@ -321,17 +412,10 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 20, "id": "c01761b3-c766-46b0-ae0b-fcf968ffb3a1", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[Stage 10:> (0 + 1) / 1]\r" - ] - }, { "name": "stdout", "output_type": "stream", @@ -339,7 +423,7 @@ "+--------------------------------------------------------------------------------+--------+----------+\n", "| sentence| label| score|\n", "+--------------------------------------------------------------------------------+--------+----------+\n", - "| |POSITIVE|0.74812096|\n", + "| |POSITIVE| 0.7481212|\n", "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pi...|NEGATIVE|0.99967253|\n", "|I watched this movie to see the direction one of the most promising young tal...|POSITIVE| 0.9994943|\n", "| This movie makes you wish imdb would let you vote a zero|NEGATIVE| 0.9981305|\n", @@ -351,25 +435,18 @@ "|This movie seems a little clunky around the edges, like not quite enough zani...|NEGATIVE|0.99965274|\n", "|I rented this movie hoping that it would provide some good entertainment and ...|NEGATIVE|0.99642426|\n", "|Well, where to start describing this celluloid debacle? You already know the ...|NEGATIVE|0.99973005|\n", - "| I hoped for this show to be somewhat realistic|POSITIVE| 0.8426521|\n", + "| I hoped for this show to be somewhat realistic|POSITIVE| 0.8426496|\n", "| All I have to say is one word|NEGATIVE| 0.9784491|\n", "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy scr...|NEGATIVE| 0.99977|\n", "|This critique tells the story of 4 little friends who went to watch Angels an...|POSITIVE| 0.9942334|\n", "| This review contains a partial spoiler|NEGATIVE| 0.996191|\n", - "| I'm rather surprised that anybody found this film touching or moving|POSITIVE| 0.8392786|\n", + "| I'm rather surprised that anybody found this film touching or moving|POSITIVE| 0.8392794|\n", "| If you like bad movies (and you must to watch this one) here's a good one|POSITIVE|0.99366415|\n", "|This is really bad, the characters were bland, the story was boring, and ther...|NEGATIVE|0.99953806|\n", "+--------------------------------------------------------------------------------+--------+----------+\n", "only showing top 20 rows\n", "\n" ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] } ], "source": [ @@ -391,21 +468,21 @@ "id": "10368010-f94d-4167-91a1-2cf9ed91a2c9", "metadata": {}, "source": [ - "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments), using a conda-pack environment created as follows:\n", + "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments) with the compatible versions of Python/Numpy for Triton 24.08, using a conda-pack environment created as follows:\n", "```\n", - "conda create -n huggingface -c conda-forge python=3.8\n", - "conda activate huggingface\n", + "conda create -n huggingface-torch -c conda-forge python=3.10.0\n", + "conda activate huggingface-torch\n", "\n", - "export PYTHONUSERSITE=True\n", - "pip install conda-pack sentencepiece sentence_transformers transformers\n", + "export PYTHONNOUSERSITE=True\n", + "pip install numpy==1.26.4 conda-pack sentencepiece sentence_transformers transformers\n", "\n", - "conda-pack # huggingface.tar.gz\n", + "conda-pack # huggingface-torch.tar.gz\n", "```" ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 21, "id": "4d4be844-4b8c-47df-bd09-0c280c7ff16b", "metadata": {}, "outputs": [], @@ -420,30 +497,19 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 22, "id": "7e53df9f-43cb-4c38-b8ac-dc2cbad99815", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", - "To disable this warning, you can either:\n", - "\t- Avoid using `tokenizers` before the fork if possible\n", - "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" - ] - } - ], + "outputs": [], "source": [ "%%bash\n", "# copy custom model to expected layout for Triton\n", "rm -rf models\n", "mkdir -p models\n", - "cp -r models_config/hf_pipeline models\n", + "cp -r models_config/hf_pipeline_torch models\n", "\n", "# add custom execution environment\n", - "cp huggingface.tar.gz models" + "cp huggingface-torch.tar.gz models" ] }, { @@ -458,7 +524,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 23, "id": "144acb8e-4c08-40fc-a9ed-f721c409ee68", "metadata": {}, "outputs": [ @@ -475,7 +541,7 @@ "[True]" ] }, - "execution_count": 15, + "execution_count": 23, "metadata": {}, "output_type": "execute_result" } @@ -497,7 +563,7 @@ " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", " else:\n", " container=client.containers.run(\n", - " \"nvcr.io/nvidia/tritonserver:23.04-py3\", \"tritonserver --model-repository=/models\",\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", " detach=True,\n", " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", " environment=[\n", @@ -544,7 +610,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 24, "id": "d53fb283-bf9e-4571-8c68-b75a41f1f067", "metadata": {}, "outputs": [], @@ -559,7 +625,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 25, "id": "29b0cc0d-c480-4e4a-bd41-207dc314cba5", "metadata": {}, "outputs": [], @@ -569,7 +635,7 @@ " import tritonclient.grpc as grpcclient\n", " \n", " np_types = {\n", - " \"BOOL\": np.dtype(np.bool8),\n", + " \"BOOL\": np.dtype(np.bool_),\n", " \"INT8\": np.dtype(np.int8),\n", " \"INT16\": np.dtype(np.int16),\n", " \"INT32\": np.dtype(np.int32),\n", @@ -609,14 +675,14 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 26, "id": "3930cfcd-3284-4c6a-a9b5-36b8053fe899", "metadata": {}, "outputs": [], "source": [ "from functools import partial\n", "\n", - "classify = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"hf_pipeline\"),\n", + "classify = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"hf_pipeline_torch\"),\n", " return_type=StructType([\n", " StructField(\"label\", StringType(), True),\n", " StructField(\"score\", FloatType(), True)\n", @@ -627,7 +693,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 27, "id": "8eecbf23-4e9e-4d4c-8645-98209b25db2c", "metadata": {}, "outputs": [ @@ -635,15 +701,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 13:> (0 + 1) / 1]\r" + "[Stage 20:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 25.5 ms, sys: 0 ns, total: 25.5 ms\n", - "Wall time: 5.42 s\n" + "CPU times: user 5.89 ms, sys: 5.41 ms, total: 11.3 ms\n", + "Wall time: 1.98 s\n" ] }, { @@ -664,7 +730,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 28, "id": "566ba28c-0ca4-4479-a24a-c8a362228b89", "metadata": {}, "outputs": [ @@ -672,15 +738,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 14:> (0 + 1) / 1]\r" + "[Stage 21:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 11.6 ms, sys: 7.68 ms, total: 19.3 ms\n", - "Wall time: 4.52 s\n" + "CPU times: user 5.87 ms, sys: 2.39 ms, total: 8.26 ms\n", + "Wall time: 1.87 s\n" ] }, { @@ -700,7 +766,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 29, "id": "44c7e776-08da-484a-ba07-9d6add1a0f15", "metadata": {}, "outputs": [ @@ -708,15 +774,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 15:> (0 + 1) / 1]\r" + "[Stage 22:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 13.1 ms, sys: 5.65 ms, total: 18.7 ms\n", - "Wall time: 4.51 s\n" + "CPU times: user 5.24 ms, sys: 1.13 ms, total: 6.37 ms\n", + "Wall time: 1.86 s\n" ] }, { @@ -736,59 +802,45 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 30, "id": "f61d79f8-661e-4d9e-a3aa-c0754b854603", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[Stage 16:> (0 + 1) / 1]\r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "+--------------------------------------------------------------------------------+--------+----------+\n", - "| sentence| label| score|\n", - "+--------------------------------------------------------------------------------+--------+----------+\n", - "| |POSITIVE| 0.7481212|\n", - "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pi...|NEGATIVE|0.99967253|\n", - "|I watched this movie to see the direction one of the most promising young tal...|POSITIVE| 0.9994943|\n", - "| This movie makes you wish imdb would let you vote a zero|NEGATIVE| 0.9981305|\n", - "|I never want to see this movie again!

Not only is it dreadfully ba...|NEGATIVE| 0.9988337|\n", - "|(As a note, I'd like to say that I saw this movie at my annual church camp, w...|POSITIVE| 0.9901974|\n", - "| Don't get me wrong, I love the TV series of League Of Gentlemen|POSITIVE| 0.9998311|\n", - "|Did you ever think, like after watching a horror movie with a group of friend...|POSITIVE| 0.9992779|\n", - "| Awful, awful, awful|NEGATIVE| 0.9997433|\n", - "|This movie seems a little clunky around the edges, like not quite enough zani...|NEGATIVE|0.99965274|\n", - "|I rented this movie hoping that it would provide some good entertainment and ...|NEGATIVE|0.99642426|\n", - "|Well, where to start describing this celluloid debacle? You already know the ...|NEGATIVE|0.99973005|\n", - "| I hoped for this show to be somewhat realistic|POSITIVE|0.84265035|\n", - "| All I have to say is one word|NEGATIVE| 0.9784491|\n", - "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy scr...|NEGATIVE| 0.99977|\n", - "|This critique tells the story of 4 little friends who went to watch Angels an...|POSITIVE| 0.9942334|\n", - "| This review contains a partial spoiler|NEGATIVE| 0.996191|\n", - "| I'm rather surprised that anybody found this film touching or moving|POSITIVE|0.83927685|\n", - "| If you like bad movies (and you must to watch this one) here's a good one|POSITIVE|0.99366415|\n", - "|This is really bad, the characters were bland, the story was boring, and ther...|NEGATIVE|0.99953806|\n", - "+--------------------------------------------------------------------------------+--------+----------+\n", + "+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+----------+\n", + "|sentence |label |score |\n", + "+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+----------+\n", + "| |POSITIVE|0.7481212 |\n", + "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story without the gratuitous sex scenes, either hard core or soft core, therefore reads like a public information film from the fifties, give this a wide miss, use a barge pole if you can|NEGATIVE|0.99967253|\n", + "|I watched this movie to see the direction one of the most promising young talents in movies was going |POSITIVE|0.9994943 |\n", + "|This movie makes you wish imdb would let you vote a zero |NEGATIVE|0.9981305 |\n", + "|I never want to see this movie again!

Not only is it dreadfully bad, but I can't stand seeing my hero Stan Laurel looking so old and sick |NEGATIVE|0.9988337 |\n", + "|(As a note, I'd like to say that I saw this movie at my annual church camp, where the entire youth group laughed at it |POSITIVE|0.9901974 |\n", + "|Don't get me wrong, I love the TV series of League Of Gentlemen |POSITIVE|0.9998311 |\n", + "|Did you ever think, like after watching a horror movie with a group of friends: \"Wow, this is so cool! We have got to make a splatter horror movie ourselves some day soon |POSITIVE|0.9992779 |\n", + "|Awful, awful, awful |NEGATIVE|0.9997433 |\n", + "|This movie seems a little clunky around the edges, like not quite enough zaniness was thrown it when it should have been |NEGATIVE|0.99965274|\n", + "|I rented this movie hoping that it would provide some good entertainment and some cool poker knowledge or stories |NEGATIVE|0.99642426|\n", + "|Well, where to start describing this celluloid debacle? You already know the big fat NADA passing as a plot, so let's jut point out that this is so PC it's offensive |NEGATIVE|0.99973005|\n", + "|I hoped for this show to be somewhat realistic |POSITIVE|0.8426496 |\n", + "|All I have to say is one word |NEGATIVE|0.9784491 |\n", + "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay |NEGATIVE|0.99977 |\n", + "|This critique tells the story of 4 little friends who went to watch Angels and Demons the movie on the first night it came out, even though it was a school night, because \"Angels and Demons is worth it |POSITIVE|0.9942334 |\n", + "|This review contains a partial spoiler |NEGATIVE|0.996191 |\n", + "|I'm rather surprised that anybody found this film touching or moving |POSITIVE|0.8392794 |\n", + "|If you like bad movies (and you must to watch this one) here's a good one |POSITIVE|0.99366415|\n", + "|This is really bad, the characters were bland, the story was boring, and there is no sex scene |NEGATIVE|0.99953806|\n", + "+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+--------+----------+\n", "only showing top 20 rows\n", "\n" ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] } ], "source": [ - "preds.show(truncate=80)" + "preds.show(truncate=False)" ] }, { @@ -803,7 +855,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 31, "id": "425d3b28-7705-45ba-8a18-ad34fc895219", "metadata": {}, "outputs": [ @@ -820,7 +872,7 @@ "[True]" ] }, - "execution_count": 23, + "execution_count": 31, "metadata": {}, "output_type": "execute_result" } @@ -844,7 +896,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 32, "id": "9f19643c-4ee4-44f2-b762-2078c0c8eba9", "metadata": {}, "outputs": [], @@ -863,7 +915,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "spark-dl-torch", "language": "python", "name": "python3" }, @@ -877,7 +929,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.10" + "version": "3.11.9" } }, "nbformat": 4, diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/sentence_transformers.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/sentence_transformers_torch.ipynb similarity index 50% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/sentence_transformers.ipynb rename to examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/sentence_transformers_torch.ipynb index cb4b5d867..4a8a04078 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/sentence_transformers.ipynb +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/huggingface/sentence_transformers_torch.ipynb @@ -6,7 +6,7 @@ "metadata": {}, "source": [ "# PySpark Huggingface Inferencing\n", - "### Sentence Transformers\n", + "## Sentence Transformers with PyTorch\n", "\n", "From: https://huggingface.co/sentence-transformers" ] @@ -21,8 +21,10 @@ "name": "stderr", "output_type": "stream", "text": [ - "/home/leey/.pyenv/versions/3.9.10/envs/spark_rapids_examples/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n", - " from .autonotebook import tqdm as notebook_tqdm\n" + "/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/sentence_transformers/cross_encoder/CrossEncoder.py:13: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n", + " from tqdm.autonotebook import tqdm, trange\n", + "/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/transformers/tokenization_utils_base.py:1617: FutureWarning: `clean_up_tokenization_spaces` was not set. It will be set to `True` by default. This behavior will be deprecated in transformers v4.45, and will be then set to `False` by default. For more details check this issue: https://github.com/huggingface/transformers/issues/31884\n", + " warnings.warn(\n" ] } ], @@ -45,145 +47,18 @@ "metadata": {}, "outputs": [ { - "data": { - "text/plain": [ - "array([[-1.76214516e-01, 1.20601304e-01, -2.93624043e-01,\n", - " -2.29858175e-01, -8.22924003e-02, 2.37709180e-01,\n", - " 3.39985013e-01, -7.80964136e-01, 1.18127793e-01,\n", - " 1.63374111e-01, -1.37715325e-01, 2.40282863e-01,\n", - " 4.25125599e-01, 1.72417879e-01, 1.05279416e-01,\n", - " 5.18164098e-01, 6.22219592e-02, 3.99285942e-01,\n", - " -1.81652382e-01, -5.85578799e-01, 4.49718162e-02,\n", - " -1.72750458e-01, -2.68443376e-01, -1.47386298e-01,\n", - " -1.89217895e-01, 1.92150623e-01, -3.83842528e-01,\n", - " -3.96006793e-01, 4.30648834e-01, -3.15320045e-01,\n", - " 3.65949810e-01, 6.05160184e-02, 3.57326001e-01,\n", - " 1.59736484e-01, -3.00984085e-01, 2.63250291e-01,\n", - " -3.94310981e-01, 1.84855387e-01, -3.99549156e-01,\n", - " -2.67889678e-01, -5.45117497e-01, -3.13403830e-02,\n", - " -4.30644304e-01, 1.33278236e-01, -1.74793929e-01,\n", - " -4.35465544e-01, -4.77378905e-01, 7.12556019e-02,\n", - " -7.37000927e-02, 5.69136977e-01, -2.82579631e-01,\n", - " 5.24974912e-02, -8.20008039e-01, 1.98296875e-01,\n", - " 1.69511944e-01, 2.71780223e-01, 2.64610887e-01,\n", - " -2.55740248e-02, -1.74096078e-01, 1.63314238e-01,\n", - " -3.95261019e-01, -3.17557529e-02, -2.62556016e-01,\n", - " 3.52754653e-01, 3.01434726e-01, -1.47197261e-01,\n", - " 2.10075721e-01, -1.84010327e-01, -4.12896097e-01,\n", - " 4.14775908e-01, -1.89769432e-01, -1.35482103e-01,\n", - " -3.79272342e-01, -4.68020253e-02, -3.33600566e-02,\n", - " 9.00392979e-02, -3.30133080e-01, -3.87317017e-02,\n", - " 3.75082225e-01, -1.46996513e-01, 4.34959859e-01,\n", - " 5.38325727e-01, -2.65445322e-01, 1.64445966e-01,\n", - " 4.17078137e-01, -4.72507551e-02, -7.48731717e-02,\n", - " -4.26260680e-01, -1.96994469e-01, 6.10315353e-02,\n", - " -4.74262595e-01, -6.48334563e-01, 3.71462375e-01,\n", - " 2.50956744e-01, 1.22529656e-01, 8.88765603e-02,\n", - " -1.06724449e-01, 5.33984527e-02, 9.74504799e-02,\n", - " -3.46659198e-02, -1.02882944e-01, 2.32289046e-01,\n", - " -2.53739715e-01, -5.13112009e-01, 1.85215965e-01,\n", - " -3.04357857e-01, -3.55209708e-02, -1.26975283e-01,\n", - " -7.71633461e-02, -5.15329778e-01, -2.28072166e-01,\n", - " 2.03343891e-02, 7.38176629e-02, -1.52558297e-01,\n", - " -4.00837600e-01, -2.47749388e-01, 3.97470415e-01,\n", - " -2.60260761e-01, 2.50905871e-01, 1.68229103e-01,\n", - " 1.33900389e-01, -2.10832264e-02, -4.70035672e-01,\n", - " 4.78850305e-01, 2.80345857e-01, -4.64546710e-01,\n", - " 3.21746945e-01, 2.34207466e-01, 2.45772362e-01,\n", - " -4.71482247e-01, 5.00401437e-01, 4.10190284e-01,\n", - " 5.15217066e-01, 2.62549222e-01, 2.11592801e-02,\n", - " -3.89687479e-01, -2.41742760e-01, -2.14834422e-01,\n", - " -8.62650797e-02, -1.65323481e-01, -5.21896258e-02,\n", - " 3.41875046e-01, 4.50314254e-01, -3.06973517e-01,\n", - " -2.02294275e-01, 6.85521781e-01, -5.33892572e-01,\n", - " 3.58471453e-01, 1.45286813e-01, -7.07055628e-02,\n", - " -1.50529206e-01, -8.56279060e-02, -7.67850205e-02,\n", - " 1.89544708e-01, -1.04067393e-01, 5.33544004e-01,\n", - " -5.27887166e-01, 2.42331959e-02, -2.64348119e-01,\n", - " -2.23186791e-01, -3.81208628e-01, 7.59914368e-02,\n", - " -4.64485019e-01, -3.36549103e-01, 4.21229810e-01,\n", - " 1.07479259e-01, 1.90457568e-01, 2.89495080e-03,\n", - " -1.08513527e-01, 1.53545514e-01, 3.16023558e-01,\n", - " -2.70837210e-02, -5.40594459e-01, 8.97289440e-02,\n", - " -1.15549557e-01, 3.97803813e-01, -4.97683465e-01,\n", - " -2.84893245e-01, 4.99861389e-02, 3.61279517e-01,\n", - " 6.90535426e-01, 1.46821350e-01, 1.73396409e-01,\n", - " -1.74582213e-01, -3.15702498e-01, 6.72999024e-02,\n", - " 2.17250124e-01, 9.78534073e-02, -1.29472524e-01,\n", - " -1.86929733e-01, 1.34877980e-01, -1.53885141e-01,\n", - " 7.44716451e-02, -1.85536250e-01, -2.80628234e-01,\n", - " -1.14144124e-01, 4.12249714e-01, 6.39493242e-02,\n", - " -1.45715356e-01, -9.82063636e-02, -1.33081853e-01,\n", - " -1.88410729e-01, -2.84840688e-02, -3.49510685e-02,\n", - " 3.34260389e-02, 6.98895752e-02, 1.90354556e-01,\n", - " -2.96724111e-01, 2.64700665e-03, 1.09140664e-01,\n", - " 1.70893949e-02, 2.60589182e-01, 3.29038322e-01,\n", - " -6.61561564e-02, 2.39665493e-01, -2.26194724e-01,\n", - " -3.36869434e-02, 1.49400219e-01, -3.21265519e-01,\n", - " -2.68578082e-01, 5.72631419e-01, -4.92308617e-01,\n", - " 2.00666517e-01, -3.49261880e-01, -2.89886966e-02,\n", - " 6.09010518e-01, -5.72333217e-01, 2.35000581e-01,\n", - " 6.47170283e-03, -3.14947553e-02, 2.78106760e-02,\n", - " -3.90340686e-01, -2.08949789e-01, -3.04452747e-01,\n", - " -7.20199049e-02, -8.29839855e-02, 3.73792797e-01,\n", - " 7.38939270e-02, -2.21075043e-02, 9.88139212e-02,\n", - " -1.51426777e-01, -1.40430599e-01, 2.26017937e-01,\n", - " 2.76090086e-01, -8.87750760e-02, -1.12816244e-01,\n", - " -2.66285956e-01, 2.77834475e-01, -4.75612208e-02,\n", - " 6.71006441e-02, -2.78584342e-02, -2.39992719e-02,\n", - " 2.51708895e-01, 4.68793869e-01, -5.39325416e-01,\n", - " 1.10598333e-01, -3.44947278e-01, 4.15990025e-01,\n", - " 7.28482902e-02, -3.19647491e-01, 4.90374297e-01,\n", - " -7.30326539e-03, -2.64258590e-03, 9.63711143e-01,\n", - " 3.23884934e-01, -7.79617876e-02, -2.37589255e-01,\n", - " 2.34038249e-01, -3.16054285e-01, -1.65644684e-03,\n", - " -1.09070671e+00, 3.38409394e-01, 4.70604822e-02,\n", - " 1.07435532e-01, -2.06672445e-01, 4.26434958e-03,\n", - " -1.38471671e-03, -5.31455398e-01, -2.75648385e-01,\n", - " -1.64648548e-01, -3.42916548e-01, -4.26118731e-01,\n", - " 6.01812005e-01, 4.55971926e-01, -2.72701979e-01,\n", - " -3.45803909e-02, 2.62752384e-01, -6.34182245e-03,\n", - " 2.79631168e-01, -2.53559083e-01, -1.68626398e-01,\n", - " 3.82935070e-02, 2.07763270e-01, -4.31525737e-01,\n", - " -7.24000186e-02, -1.26854718e-01, 2.07032599e-02,\n", - " 5.74441671e-01, 3.54672760e-01, 9.28299800e-02,\n", - " 6.70504868e-02, 1.11520380e-01, -1.86511762e-02,\n", - " 4.62352008e-01, 2.72504658e-01, -3.60473931e-01,\n", - " 5.29415190e-01, -1.00307481e-03, -8.81362036e-02,\n", - " 1.49975210e-01, 5.25863320e-02, 4.63517606e-01,\n", - " -3.96831453e-01, 2.42640764e-01, -2.08912343e-01,\n", - " 3.65672171e-01, -4.73377790e-04, 5.33963263e-01,\n", - " -1.97879702e-01, 3.11582834e-01, -6.96714938e-01,\n", - " -4.29500610e-01, -4.49359357e-01, -2.71370225e-02,\n", - " -6.98709935e-02, 2.06174642e-01, -1.57107607e-01,\n", - " 4.43521231e-01, -6.74267113e-02, -3.00924242e-01,\n", - " 5.14859617e-01, 3.36029500e-01, 6.63374960e-02,\n", - " -1.15235247e-01, -2.95980442e-02, 2.79471934e-01,\n", - " -3.48198377e-02, -7.29323775e-02, -4.58472818e-02,\n", - " 1.54262766e-01, 8.09356093e-01, 5.20328283e-01,\n", - " -4.02114809e-01, -3.23153809e-02, -1.10363849e-01,\n", - " 7.50504881e-02, -1.51098818e-01, 8.45739901e-01,\n", - " -1.80844069e-01, 3.22573632e-01, 1.04708232e-01,\n", - " 3.19663674e-01, -1.55085340e-01, 1.69236794e-01,\n", - " -2.56996810e-01, 2.01208934e-01, 1.77392989e-01,\n", - " -2.74333209e-01, -3.36944401e-01, 5.02356768e-01,\n", - " -1.18357144e-01, -2.01166883e-01, -5.36485732e-01,\n", - " -7.69810155e-02, 1.15381051e-02, -2.36464351e-01,\n", - " -2.98769865e-02, 1.31366819e-01, 2.94184357e-01,\n", - " 9.90916416e-02, -5.43897390e-01, 1.40812859e-01,\n", - " 3.66998732e-01, 5.04862480e-02, 1.99122518e-01,\n", - " -2.80674607e-01, 4.34192210e-01, -1.40274912e-01,\n", - " 5.78049004e-01, 1.77715704e-01, 8.98363292e-02,\n", - " 3.29651982e-01, 6.13008998e-02, -3.24933499e-01]], dtype=float32)" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" + "name": "stdout", + "output_type": "stream", + "text": [ + "[-0.17621444 0.1206013 -0.29362372 -0.22985819 -0.08229247 0.2377093\n", + " 0.33998525 -0.7809643 0.11812777 0.16337365 -0.13771524 0.24028276\n", + " 0.4251256 0.17241786 0.10527937 0.5181643 0.062222 0.39928585\n", + " -0.18165241 -0.58557856]\n" + ] } ], "source": [ - "embedding" + "print(embedding[0][:20])" ] }, { @@ -212,41 +87,105 @@ "source": [ "from pyspark.ml.functions import predict_batch_udf\n", "from pyspark.sql.functions import col, struct\n", - "from pyspark.sql.types import ArrayType, FloatType" + "from pyspark.sql.types import ArrayType, FloatType\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf\n", + "from datasets import load_dataset" ] }, { "cell_type": "code", "execution_count": 4, - "id": "836e5f84-12c6-4c95-838e-53de7e46a20b", + "id": "23ec67ba", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - " \r" + "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", + "To disable this warning, you can either:\n", + "\t- Avoid using `tokenizers` before the fork if possible\n", + "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n", + "24/10/08 00:19:28 WARN Utils: Your hostname, cb4ae00-lcedt resolves to a loopback address: 127.0.1.1; using 10.110.47.100 instead (on interface eno1)\n", + "24/10/08 00:19:28 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n", + "Setting default log level to \"WARN\".\n", + "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n", + "24/10/08 00:19:28 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n" ] } ], "source": [ - "# only use first N examples, since this is slow\n", - "df = spark.read.parquet(\"imdb_test\").limit(100).cache()" + "import os\n", + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" ] }, { "cell_type": "code", "execution_count": 5, - "id": "36703d23-37a3-40df-b09a-c68206d285b6", + "id": "9bc1edb5", "metadata": {}, "outputs": [ { "name": "stderr", "output_type": "stream", "text": [ - "[Stage 1:==============================================> (8 + 2) / 10]\r" + " \r" ] - }, + } + ], + "source": [ + "# load IMDB reviews (test) dataset and write to parquet\n", + "data = load_dataset(\"imdb\", split=\"test\")\n", + "\n", + "lines = []\n", + "for example in data:\n", + " lines.append([example[\"text\"].split(\".\")[0]])\n", + "\n", + "len(lines)\n", + "\n", + "df = spark.createDataFrame(lines, ['lines']).repartition(10)\n", + "df.schema\n", + "\n", + "df.write.mode(\"overwrite\").parquet(\"imdb_test\")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "836e5f84-12c6-4c95-838e-53de7e46a20b", + "metadata": {}, + "outputs": [], + "source": [ + "# only use first N examples, since this is slow\n", + "df = spark.read.parquet(\"imdb_test\").limit(100).cache()" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "36703d23-37a3-40df-b09a-c68206d285b6", + "metadata": {}, + "outputs": [ { "name": "stdout", "output_type": "stream", @@ -254,37 +193,30 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| lines|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "|...But not this one! I always wanted to know \"what happened\" next. We will never know for sure what happened because ...|\n", - "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story without the gratuitous sex scen...|\n", - "|I watched this movie to see the direction one of the most promising young talents in movies was going. Unfortunately,...|\n", - "|This movie makes you wish imdb would let you vote a zero. One of the two movies I've ever walked out of. It's very ha...|\n", - "|I never want to see this movie again!

Not only is it dreadfully bad, but I can't stand seeing my hero Stan...|\n", - "|(As a note, I'd like to say that I saw this movie at my annual church camp, where the entire youth group laughed at i...|\n", - "|Don't get me wrong, I love the TV series of League Of Gentlemen. It was funny, twisted and completely inspired. I was...|\n", - "|Did you ever think, like after watching a horror movie with a group of friends: \"Wow, this is so cool! We have got to...|\n", - "|Awful, awful, awful...

I loved the original film. It was funny, charming, and had heart... this piece of j...|\n", - "|This movie seems a little clunky around the edges, like not quite enough zaniness was thrown it when it should have b...|\n", - "|I rented this movie hoping that it would provide some good entertainment and some cool poker knowledge or stories. Wh...|\n", - "|Well, where to start describing this celluloid debacle? You already know the big fat NADA passing as a plot, so let's...|\n", - "|I hoped for this show to be somewhat realistic. It stroke me as just another mainstream show after I watched it. I di...|\n", - "|All I have to say is one word...SUCKS!!!!. The only reason I gave this a 2 is because Josh Hartnett was in it and he'...|\n", - "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay.

The lighting at is s...|\n", - "|This critique tells the story of 4 little friends who went to watch Angels and Demons the movie on the first night it...|\n", - "|This review contains a partial spoiler.

Shallow from the outset, 'D.O.A.' at least starts as if it might b...|\n", - "|I'm rather surprised that anybody found this film touching or moving.

The basic premise of the film sounde...|\n", - "|If you like bad movies (and you must to watch this one) here's a good one. Not quite as funny as the first, but much ...|\n", - "|This is really bad, the characters were bland, the story was boring, and there is no sex scene. Furthermore, it lacks...|\n", + "| This is so overly clichéd you'll want to switch it off after the first 45 minutes|\n", + "| I was very disappointed by this movie|\n", + "| I think vampire movies (usually) are wicked|\n", + "| Though not a complete waste of time, 'Eighteen' really wasn't all sweet as it pretended to be|\n", + "|This film did well at the box office, and the producers of this mess thought the stars had such good chemistry in thi...|\n", + "| Peter Crawford discovers a comet on a collision course with the moon|\n", + "|This tale of the upper-classes getting their come-uppance and wallowing in their high-class misery is like a contempo...|\n", + "|Words almost fail me to describe how terrible this Irish vanity project (funded by Canadian taxpayers - both federal ...|\n", + "| This was the most uninteresting horror flick I have seen to date|\n", + "| Heart of Darkness was terrible|\n", + "| I saw this movie when it was first released in Pittsburgh Pa|\n", + "|It was funny because the whole thing was so unrealistic, I mean, come on, like a pop star would just show up at a pub...|\n", + "|Watching this movie, you just have to ask: What were they thinking? There are so many noticeably bad parts of this mo...|\n", + "| In a sense, this movie did not even compare to the novel|\n", + "| Poor Jane Austen ought to be glad she's not around to see this dreadful wreck of an adaptation|\n", + "| I gave this movie a four-star rating for a few reasons|\n", + "| It seems that Dee Snyder ran out of ideas halfway through the script|\n", + "| Now, let me see if I have this correct, a lunatic serial killer is going around murdering estate agents|\n", + "| Tommy Lee Jones was the best Woodroe and no one can play Woodroe F|\n", + "|First of all, I would like to say that I am a fan of all of the actors that appear in this film and at the time that ...|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] } ], "source": [ @@ -293,7 +225,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 8, "id": "f780c026-0f3f-4aea-8b61-5b3dbae83fb7", "metadata": {}, "outputs": [], @@ -308,7 +240,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 9, "id": "f5c88ddc-ca19-4430-8b0e-b9fae143b237", "metadata": {}, "outputs": [], @@ -320,7 +252,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 10, "id": "85344c22-4a4d-4cb0-8771-5836ae2794db", "metadata": {}, "outputs": [ @@ -328,15 +260,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 4:> (0 + 1) / 1]\r" + "[Stage 9:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 17.8 ms, sys: 9.35 ms, total: 27.2 ms\n", - "Wall time: 8.36 s\n" + "CPU times: user 4.34 ms, sys: 4.15 ms, total: 8.48 ms\n", + "Wall time: 2.58 s\n" ] }, { @@ -356,7 +288,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 11, "id": "c23bb885-6ab0-4471-943d-4c10414100fa", "metadata": {}, "outputs": [ @@ -364,15 +296,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 6:> (0 + 1) / 1]\r" + "[Stage 11:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 17.7 ms, sys: 1.13 ms, total: 18.8 ms\n", - "Wall time: 3.25 s\n" + "CPU times: user 1.76 ms, sys: 4.89 ms, total: 6.65 ms\n", + "Wall time: 2.47 s\n" ] }, { @@ -391,7 +323,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 12, "id": "93bc6da3-d853-4233-b805-cb4a46f4f9b9", "metadata": {}, "outputs": [ @@ -399,15 +331,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 8:> (0 + 1) / 1]\r" + "[Stage 13:> (0 + 1) / 1]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 16.9 ms, sys: 0 ns, total: 16.9 ms\n", - "Wall time: 2.91 s\n" + "CPU times: user 1.55 ms, sys: 6.05 ms, total: 7.6 ms\n", + "Wall time: 2.46 s\n" ] }, { @@ -426,10 +358,17 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 13, "id": "2073616f-7151-4760-92f2-441dd0bfe9fe", "metadata": {}, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 15:> (0 + 1) / 1]\r" + ] + }, { "name": "stdout", "output_type": "stream", @@ -437,30 +376,37 @@ "+------------------------------------------------------------+------------------------------------------------------------+\n", "| lines| encoding|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", - "|...But not this one! I always wanted to know \"what happen...|[0.050629966, -0.19899231, 2.686046E-4, 0.13270327, -0.16...|\n", - "|Hard up, No proper jobs going down at the pit, why not re...|[0.08634103, -0.002254839, 0.10213216, -0.03454912, -0.23...|\n", - "|I watched this movie to see the direction one of the most...|[0.008758117, -0.0083419345, -0.119090386, 0.025434377, -...|\n", - "|This movie makes you wish imdb would let you vote a zero....|[0.24080081, -0.14614257, 0.18119521, 0.118741795, 0.1022...|\n", - "|I never want to see this movie again!

Not only...|[0.32271573, -0.14145091, 0.09245593, 0.04562203, 0.07219...|\n", - "|(As a note, I'd like to say that I saw this movie at my a...|[0.1308969, 0.14792246, -0.021109976, -0.16882573, -0.055...|\n", - "|Don't get me wrong, I love the TV series of League Of Gen...|[-0.19420472, 0.116419405, 0.01985946, -0.37481546, 0.052...|\n", - "|Did you ever think, like after watching a horror movie wi...|[0.050077364, -0.34728476, -0.47222477, 0.09191189, -0.16...|\n", - "|Awful, awful, awful...

I loved the original fi...|[-0.23921771, -0.22389278, -0.0042956644, 0.058358684, 0....|\n", - "|This movie seems a little clunky around the edges, like n...|[-0.12948105, -0.16344212, -0.28761974, -0.10628598, -0.0...|\n", - "|I rented this movie hoping that it would provide some goo...|[-0.030982, -0.13821997, 0.14594209, -0.20565805, -0.0225...|\n", - "|Well, where to start describing this celluloid debacle? Y...|[-0.29094818, 0.026240889, -0.21248402, 0.028826537, -0.1...|\n", - "|I hoped for this show to be somewhat realistic. It stroke...|[0.14481407, -0.13123356, -0.47293735, -0.21168816, 0.001...|\n", - "|All I have to say is one word...SUCKS!!!!. The only reaso...|[0.018178312, 0.11847291, -0.33938172, -0.15572134, 0.051...|\n", - "|Honestly awful film, bad editing, awful lighting, dire di...|[-0.10033801, -0.28231186, 0.18979141, 0.042497832, 0.125...|\n", - "|This critique tells the story of 4 little friends who wen...|[0.1318425, -0.1671868, -0.013854267, 0.14505053, -0.2534...|\n", - "|This review contains a partial spoiler.

Shallo...|[-0.08296674, -0.08548329, -0.13219479, -0.20946309, 0.01...|\n", - "|I'm rather surprised that anybody found this film touchin...|[-0.21176083, -0.12755248, -0.28217235, 0.02004116, 0.074...|\n", - "|If you like bad movies (and you must to watch this one) h...|[-0.38088003, -0.1916466, 0.16510564, -0.11013024, -0.233...|\n", - "|This is really bad, the characters were bland, the story ...|[0.09919267, 0.042636175, -0.17805319, -0.1818586, -0.123...|\n", + "|This is so overly clichéd you'll want to switch it off af...|[-0.06755405, -0.13365394, 0.36675274, -0.2772311, -0.085...|\n", + "| I was very disappointed by this movie|[-0.05903806, 0.16684641, 0.16768408, 0.10940918, 0.18100...|\n", + "| I think vampire movies (usually) are wicked|[0.025601083, -0.5308639, -0.319133, -0.013351389, -0.338...|\n", + "|Though not a complete waste of time, 'Eighteen' really wa...|[0.20991832, 0.5228605, 0.44517252, -0.031682555, -0.4117...|\n", + "|This film did well at the box office, and the producers o...|[0.18097948, -0.03622232, -0.34149718, 0.061557338, -0.06...|\n", + "|Peter Crawford discovers a comet on a collision course wi...|[-0.27548054, 0.196654, -0.24626413, -0.39380816, -0.5501...|\n", + "|This tale of the upper-classes getting their come-uppance...|[0.24201547, 0.011018356, -0.080340266, 0.31388673, -0.28...|\n", + "|Words almost fail me to describe how terrible this Irish ...|[0.055901285, -0.14539501, -0.14005454, -0.038912475, 0.4...|\n", + "|This was the most uninteresting horror flick I have seen ...|[0.27159664, -0.012541974, -0.31898177, 0.058205508, 0.56...|\n", + "| Heart of Darkness was terrible|[0.1593065, 0.36501122, 0.10715093, 0.76344764, 0.2555183...|\n", + "|I saw this movie when it was first released in Pittsburgh Pa|[-0.34647614, 0.115615666, -0.18874267, 0.36590436, -0.06...|\n", + "|It was funny because the whole thing was so unrealistic, ...|[0.09473594, -0.43785918, 0.14436111, 0.0045353747, -0.08...|\n", + "|Watching this movie, you just have to ask: What were they...|[0.43020695, -0.09714467, 0.1356213, 0.23126744, -0.03908...|\n", + "| In a sense, this movie did not even compare to the novel|[0.2838324, -0.018966805, -0.37275136, 0.27034461, 0.2017...|\n", + "|Poor Jane Austen ought to be glad she's not around to see...|[0.27462235, -0.32494685, 0.48243234, 0.07208571, 0.22470...|\n", + "| I gave this movie a four-star rating for a few reasons|[0.31143323, -0.09470663, -0.10863629, 0.077851094, -0.15...|\n", + "|It seems that Dee Snyder ran out of ideas halfway through...|[0.44354546, -0.08122106, -0.15206784, -0.29244298, 0.559...|\n", + "|Now, let me see if I have this correct, a lunatic serial ...|[0.39831734, 0.15871558, -0.35366735, -0.11643518, -0.137...|\n", + "|Tommy Lee Jones was the best Woodroe and no one can play ...|[-0.20960264, -0.15760101, -0.30596393, -0.51817703, -0.0...|\n", + "|First of all, I would like to say that I am a fan of all ...|[0.25831866, -0.26871824, 0.026099348, -0.3459879, -0.180...|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] } ], "source": [ @@ -479,24 +425,24 @@ }, { "cell_type": "markdown", - "id": "a8211920-234e-480f-bf87-6d719090e292", + "id": "5f502a20", "metadata": {}, "source": [ - "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments), using a conda-pack environment created as follows:\n", + "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments) with the compatible versions of Python/Numpy for Triton 24.08, using a conda-pack environment created as follows:\n", "```\n", - "conda create -n huggingface -c conda-forge python=3.8\n", - "conda activate huggingface\n", + "conda create -n huggingface-torch -c conda-forge python=3.10.0\n", + "conda activate huggingface-torch\n", "\n", - "export PYTHONUSERSITE=True\n", - "pip install conda-pack sentencepiece sentence_transformers transformers\n", + "export PYTHONNOUSERSITE=True\n", + "pip install numpy==1.26.4 conda-pack sentencepiece sentence_transformers transformers\n", "\n", - "conda-pack # huggingface.tar.gz\n", + "conda-pack # huggingface-torch.tar.gz\n", "```" ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 14, "id": "772e337e-1098-4c7b-ba81-8cb221a518e2", "metadata": {}, "outputs": [], @@ -510,30 +456,19 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 15, "id": "69d0c93a-bb0b-46c5-9d28-7b08a2e70964", "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...\n", - "To disable this warning, you can either:\n", - "\t- Avoid using `tokenizers` before the fork if possible\n", - "\t- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)\n" - ] - } - ], + "outputs": [], "source": [ "%%bash\n", "# copy custom model to expected layout for Triton\n", "rm -rf models\n", "mkdir -p models\n", - "cp -r models_config/hf_transformer models\n", + "cp -r models_config/hf_transformer_torch models\n", "\n", "# add custom execution environment\n", - "cp huggingface.tar.gz models" + "cp huggingface-torch.tar.gz models" ] }, { @@ -548,7 +483,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 16, "id": "1654cdc1-4f9a-4fd5-b7ac-6ca4215bde5d", "metadata": {}, "outputs": [ @@ -565,7 +500,7 @@ "[True]" ] }, - "execution_count": 14, + "execution_count": 16, "metadata": {}, "output_type": "execute_result" } @@ -587,7 +522,7 @@ " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", " else:\n", " container=client.containers.run(\n", - " \"nvcr.io/nvidia/tritonserver:23.04-py3\", \"tritonserver --model-repository=/models\",\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", " detach=True,\n", " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", " environment=[\n", @@ -629,7 +564,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 17, "id": "2969d502-e97b-49d6-bf80-7d177ae867cf", "metadata": {}, "outputs": [], @@ -642,7 +577,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 18, "id": "c8f1e6d6-6519-49e7-8465-4419547633b8", "metadata": {}, "outputs": [ @@ -650,7 +585,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "23/05/19 19:15:37 WARN CacheManager: Asked to cache already cached data.\n" + "24/10/08 00:20:24 WARN CacheManager: Asked to cache already cached data.\n" ] } ], @@ -661,7 +596,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 19, "id": "29b0cc0d-c480-4e4a-bd41-207dc314cba5", "metadata": {}, "outputs": [], @@ -671,7 +606,7 @@ " import tritonclient.grpc as grpcclient\n", " \n", " np_types = {\n", - " \"BOOL\": np.dtype(np.bool8),\n", + " \"BOOL\": np.dtype(np.bool_),\n", " \"INT8\": np.dtype(np.int8),\n", " \"INT16\": np.dtype(np.int16),\n", " \"INT32\": np.dtype(np.int32),\n", @@ -711,12 +646,12 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 20, "id": "9c712b8f-6eb4-4fb8-9f0a-04feef847fea", "metadata": {}, "outputs": [], "source": [ - "encode = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"hf_transformer\"),\n", + "encode = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"hf_transformer_torch\"),\n", " return_type=ArrayType(FloatType()),\n", " input_tensor_shapes=[[1]],\n", " batch_size=100)" @@ -724,30 +659,16 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 21, "id": "934c1a1f-b126-45b0-9c15-265236820ad3", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[Stage 14:> (0 + 1) / 1]\r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 19.7 ms, sys: 3.95 ms, total: 23.6 ms\n", - "Wall time: 2.67 s\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" + "CPU times: user 4.65 ms, sys: 2.85 ms, total: 7.49 ms\n", + "Wall time: 480 ms\n" ] } ], @@ -760,7 +681,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 22, "id": "f84cd3f6-b6a8-4142-859a-91f3c183457b", "metadata": {}, "outputs": [ @@ -768,8 +689,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 7.66 ms, sys: 3.04 ms, total: 10.7 ms\n", - "Wall time: 265 ms\n" + "CPU times: user 1.45 ms, sys: 1.1 ms, total: 2.56 ms\n", + "Wall time: 384 ms\n" ] } ], @@ -781,7 +702,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 23, "id": "921a4c01-e296-4406-be90-86f20c8c582d", "metadata": {}, "outputs": [ @@ -789,8 +710,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 12.6 ms, sys: 570 µs, total: 13.2 ms\n", - "Wall time: 261 ms\n" + "CPU times: user 1.63 ms, sys: 1.28 ms, total: 2.91 ms\n", + "Wall time: 416 ms\n" ] } ], @@ -802,7 +723,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 24, "id": "9f67584e-9c4e-474f-b6ea-7811b14d116e", "metadata": {}, "outputs": [ @@ -813,26 +734,26 @@ "+------------------------------------------------------------+------------------------------------------------------------+\n", "| lines| encoding|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", - "|...But not this one! I always wanted to know \"what happen...|[0.050629944, -0.19899224, 2.68735E-4, 0.13270333, -0.160...|\n", - "|Hard up, No proper jobs going down at the pit, why not re...|[0.08634147, -0.002254737, 0.10213226, -0.034549147, -0.2...|\n", - "|I watched this movie to see the direction one of the most...|[0.008757966, -0.008341991, -0.11909033, 0.02543464, -0.2...|\n", - "|This movie makes you wish imdb would let you vote a zero....|[0.24080098, -0.14614293, 0.1811954, 0.11874188, 0.102292...|\n", - "|I never want to see this movie again!

Not only...|[0.3227157, -0.14145078, 0.0924558, 0.045622032, 0.072197...|\n", - "|(As a note, I'd like to say that I saw this movie at my a...|[0.13089702, 0.1479226, -0.021110116, -0.16882578, -0.055...|\n", - "|Don't get me wrong, I love the TV series of League Of Gen...|[-0.19420475, 0.11641937, 0.019859463, -0.37481567, 0.052...|\n", - "|Did you ever think, like after watching a horror movie wi...|[0.050077528, -0.34728497, -0.4722248, 0.091912046, -0.16...|\n", - "|Awful, awful, awful...

I loved the original fi...|[-0.2392176, -0.22389287, -0.004295718, 0.05835876, 0.082...|\n", - "|This movie seems a little clunky around the edges, like n...|[-0.12948103, -0.16344213, -0.2876199, -0.106286034, -0.0...|\n", - "|I rented this movie hoping that it would provide some goo...|[-0.03098194, -0.13821997, 0.1459418, -0.20565815, -0.022...|\n", - "|Well, where to start describing this celluloid debacle? Y...|[-0.29094803, 0.026240645, -0.21248397, 0.028826557, -0.1...|\n", - "|I hoped for this show to be somewhat realistic. It stroke...|[0.14481404, -0.13123384, -0.47293738, -0.21168788, 0.001...|\n", - "|All I have to say is one word...SUCKS!!!!. The only reaso...|[0.018178271, 0.11847262, -0.3393819, -0.15572123, 0.0515...|\n", - "|Honestly awful film, bad editing, awful lighting, dire di...|[-0.10033788, -0.28231215, 0.18979158, 0.042498272, 0.125...|\n", - "|This critique tells the story of 4 little friends who wen...|[0.13184246, -0.16718695, -0.013854082, 0.14505032, -0.25...|\n", - "|This review contains a partial spoiler.

Shallo...|[-0.08296674, -0.08548302, -0.1321949, -0.20946284, 0.010...|\n", - "|I'm rather surprised that anybody found this film touchin...|[-0.21176091, -0.1275524, -0.28217238, 0.020041106, 0.074...|\n", - "|If you like bad movies (and you must to watch this one) h...|[-0.38088012, -0.19164667, 0.16510546, -0.11012972, -0.23...|\n", - "|This is really bad, the characters were bland, the story ...|[0.099192545, 0.04263605, -0.17805345, -0.18185869, -0.12...|\n", + "|This is so overly clichéd you'll want to switch it off af...|[-0.06755393, -0.1336537, 0.366753, -0.2772312, -0.085145...|\n", + "| I was very disappointed by this movie|[-0.059038587, 0.1668467, 0.16768396, 0.10940957, 0.18100...|\n", + "| I think vampire movies (usually) are wicked|[0.025601566, -0.5308643, -0.31913283, -0.013350786, -0.3...|\n", + "|Though not a complete waste of time, 'Eighteen' really wa...|[0.2099183, 0.5228606, 0.4451728, -0.031682458, -0.411756...|\n", + "|This film did well at the box office, and the producers o...|[0.1809797, -0.036222238, -0.34149715, 0.06155738, -0.066...|\n", + "|Peter Crawford discovers a comet on a collision course wi...|[-0.27548066, 0.196654, -0.24626443, -0.3938084, -0.55015...|\n", + "|This tale of the upper-classes getting their come-uppance...|[0.24201535, 0.011018419, -0.080340445, 0.31388694, -0.28...|\n", + "|Words almost fail me to describe how terrible this Irish ...|[0.05590127, -0.14539507, -0.14005487, -0.03891221, 0.444...|\n", + "|This was the most uninteresting horror flick I have seen ...|[0.2715968, -0.012542339, -0.3189819, 0.05820581, 0.56001...|\n", + "| Heart of Darkness was terrible|[0.15930629, 0.36501077, 0.10715161, 0.7634482, 0.2555183...|\n", + "|I saw this movie when it was first released in Pittsburgh Pa|[-0.34647676, 0.11561544, -0.18874292, 0.36590466, -0.068...|\n", + "|It was funny because the whole thing was so unrealistic, ...|[0.09473588, -0.4378593, 0.14436121, 0.0045354995, -0.085...|\n", + "|Watching this movie, you just have to ask: What were they...|[0.43020678, -0.09714476, 0.13562134, 0.23126753, -0.0390...|\n", + "| In a sense, this movie did not even compare to the novel|[0.28383228, -0.01896684, -0.37275153, 0.27034503, 0.2017...|\n", + "|Poor Jane Austen ought to be glad she's not around to see...|[0.27462238, -0.32494652, 0.48243237, 0.07208576, 0.22470...|\n", + "| I gave this movie a four-star rating for a few reasons|[0.311433, -0.09470633, -0.10863638, 0.07785072, -0.15611...|\n", + "|It seems that Dee Snyder ran out of ideas halfway through...|[0.44354525, -0.08122053, -0.15206799, -0.29244322, 0.559...|\n", + "|Now, let me see if I have this correct, a lunatic serial ...|[0.39831725, 0.15871589, -0.35366756, -0.11643555, -0.137...|\n", + "|Tommy Lee Jones was the best Woodroe and no one can play ...|[-0.20960276, -0.157601, -0.30596414, -0.5181772, -0.0852...|\n", + "|First of all, I would like to say that I am a fan of all ...|[0.25831848, -0.26871827, 0.026099432, -0.34598774, -0.18...|\n", "+------------------------------------------------------------+------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n" @@ -855,7 +776,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 25, "id": "d8e5466b-b5dc-4fe1-9012-0c87cdd72962", "metadata": {}, "outputs": [ @@ -872,7 +793,7 @@ "[True]" ] }, - "execution_count": 23, + "execution_count": 25, "metadata": {}, "output_type": "execute_result" } @@ -896,7 +817,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 26, "id": "e82b9518-da7b-4ebc-8990-c8ab909bec18", "metadata": {}, "outputs": [], @@ -915,7 +836,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "spark-dl-torch", "language": "python", "name": "python3" }, @@ -929,7 +850,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.10" + "version": "3.11.9" } }, "nbformat": 4, diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/image_classification.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/image_classification.ipynb deleted file mode 100644 index 8d685eff5..000000000 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/image_classification.ipynb +++ /dev/null @@ -1,2180 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "9e87c927", - "metadata": {}, - "source": [ - "# PySpark PyTorch Inference\n", - "\n", - "### Image Classification\n", - "Based on: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "91d7ec98", - "metadata": {}, - "outputs": [], - "source": [ - "import torch\n", - "\n", - "from torch import nn\n", - "from torch.utils.data import DataLoader\n", - "from torchvision import datasets\n", - "from torchvision.transforms import ToTensor" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "d714f40d", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "'2.0.1+cpu'" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "torch.__version__" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "1c942a46", - "metadata": {}, - "outputs": [], - "source": [ - "# Download training data from open datasets.\n", - "training_data = datasets.FashionMNIST(\n", - " root=\"data\",\n", - " train=True,\n", - " download=True,\n", - " transform=ToTensor(),\n", - ")\n", - "\n", - "# Download test data from open datasets.\n", - "test_data = datasets.FashionMNIST(\n", - " root=\"data\",\n", - " train=False,\n", - " download=True,\n", - " transform=ToTensor(),\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "4a89aa8e-ef62-4aac-8260-4b004f2c1b55", - "metadata": {}, - "outputs": [], - "source": [ - "classes = [\n", - " \"T-shirt/top\",\n", - " \"Trouser\",\n", - " \"Pullover\",\n", - " \"Dress\",\n", - " \"Coat\",\n", - " \"Sandal\",\n", - " \"Shirt\",\n", - " \"Sneaker\",\n", - " \"Bag\",\n", - " \"Ankle boot\",\n", - "]" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "10a97111", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Shape of X [N, C, H, W]: torch.Size([64, 1, 28, 28]) torch.float32\n", - "Shape of y: torch.Size([64]) torch.int64\n" - ] - } - ], - "source": [ - "batch_size = 64\n", - "\n", - "# Create data loaders.\n", - "train_dataloader = DataLoader(training_data, batch_size=batch_size)\n", - "test_dataloader = DataLoader(test_data, batch_size=batch_size)\n", - "\n", - "for X, y in test_dataloader:\n", - " print(f\"Shape of X [N, C, H, W]: {X.shape} {X.dtype}\")\n", - " print(f\"Shape of y: {y.shape} {y.dtype}\")\n", - " break" - ] - }, - { - "cell_type": "markdown", - "id": "ca7af350", - "metadata": {}, - "source": [ - "### Create model" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "512d0bc7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Using cpu device\n", - "NeuralNetwork(\n", - " (flatten): Flatten(start_dim=1, end_dim=-1)\n", - " (linear_relu_stack): Sequential(\n", - " (0): Linear(in_features=784, out_features=512, bias=True)\n", - " (1): ReLU()\n", - " (2): Linear(in_features=512, out_features=512, bias=True)\n", - " (3): ReLU()\n", - " (4): Linear(in_features=512, out_features=10, bias=True)\n", - " )\n", - ")\n" - ] - } - ], - "source": [ - "# Get cpu or gpu device for training.\n", - "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", - "print(f\"Using {device} device\")\n", - "\n", - "# Define model\n", - "class NeuralNetwork(nn.Module):\n", - " def __init__(self):\n", - " super(NeuralNetwork, self).__init__()\n", - " self.flatten = nn.Flatten()\n", - " self.linear_relu_stack = nn.Sequential(\n", - " nn.Linear(28*28, 512),\n", - " nn.ReLU(),\n", - " nn.Linear(512, 512),\n", - " nn.ReLU(),\n", - " nn.Linear(512, 10)\n", - " )\n", - "\n", - " def forward(self, x):\n", - " x = self.flatten(x)\n", - " logits = self.linear_relu_stack(x)\n", - " return logits\n", - "\n", - "model = NeuralNetwork().to(device)\n", - "print(model)" - ] - }, - { - "cell_type": "markdown", - "id": "4573c1b7", - "metadata": {}, - "source": [ - "### Train Model" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "4d4f5538", - "metadata": {}, - "outputs": [], - "source": [ - "loss_fn = nn.CrossEntropyLoss()\n", - "optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "92d9076a", - "metadata": {}, - "outputs": [], - "source": [ - "def train(dataloader, model, loss_fn, optimizer):\n", - " size = len(dataloader.dataset)\n", - " model.train()\n", - " for batch, (X, y) in enumerate(dataloader):\n", - " X, y = X.to(device), y.to(device)\n", - "\n", - " # Compute prediction error\n", - " pred = model(X)\n", - " loss = loss_fn(pred, y)\n", - "\n", - " # Backpropagation\n", - " optimizer.zero_grad()\n", - " loss.backward()\n", - " optimizer.step()\n", - "\n", - " if batch % 100 == 0:\n", - " loss, current = loss.item(), batch * len(X)\n", - " print(f\"loss: {loss:>7f} [{current:>5d}/{size:>5d}]\")" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "11c5650d", - "metadata": {}, - "outputs": [], - "source": [ - "def test(dataloader, model, loss_fn):\n", - " size = len(dataloader.dataset)\n", - " num_batches = len(dataloader)\n", - " model.eval()\n", - " test_loss, correct = 0, 0\n", - " with torch.no_grad():\n", - " for X, y in dataloader:\n", - " X, y = X.to(device), y.to(device)\n", - " pred = model(X)\n", - " test_loss += loss_fn(pred, y).item()\n", - " correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n", - " test_loss /= num_batches\n", - " correct /= size\n", - " print(f\"Test Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \\n\")" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "854608e6", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Epoch 1\n", - "-------------------------------\n", - "loss: 2.305429 [ 0/60000]\n", - "loss: 2.297022 [ 6400/60000]\n", - "loss: 2.266470 [12800/60000]\n", - "loss: 2.265714 [19200/60000]\n", - "loss: 2.253563 [25600/60000]\n", - "loss: 2.203140 [32000/60000]\n", - "loss: 2.224580 [38400/60000]\n", - "loss: 2.182435 [44800/60000]\n", - "loss: 2.177402 [51200/60000]\n", - "loss: 2.141706 [57600/60000]\n", - "Test Error: \n", - " Accuracy: 20.4%, Avg loss: 2.142049 \n", - "\n", - "Epoch 2\n", - "-------------------------------\n", - "loss: 2.150640 [ 0/60000]\n", - "loss: 2.147359 [ 6400/60000]\n", - "loss: 2.082054 [12800/60000]\n", - "loss: 2.101886 [19200/60000]\n", - "loss: 2.044116 [25600/60000]\n", - "loss: 1.972289 [32000/60000]\n", - "loss: 2.005282 [38400/60000]\n", - "loss: 1.923009 [44800/60000]\n", - "loss: 1.926389 [51200/60000]\n", - "loss: 1.847295 [57600/60000]\n", - "Test Error: \n", - " Accuracy: 54.4%, Avg loss: 1.854991 \n", - "\n", - "Epoch 3\n", - "-------------------------------\n", - "loss: 1.888080 [ 0/60000]\n", - "loss: 1.862951 [ 6400/60000]\n", - "loss: 1.743250 [12800/60000]\n", - "loss: 1.784036 [19200/60000]\n", - "loss: 1.660822 [25600/60000]\n", - "loss: 1.621519 [32000/60000]\n", - "loss: 1.635365 [38400/60000]\n", - "loss: 1.545240 [44800/60000]\n", - "loss: 1.565089 [51200/60000]\n", - "loss: 1.461044 [57600/60000]\n", - "Test Error: \n", - " Accuracy: 60.3%, Avg loss: 1.486240 \n", - "\n", - "Epoch 4\n", - "-------------------------------\n", - "loss: 1.551637 [ 0/60000]\n", - "loss: 1.524140 [ 6400/60000]\n", - "loss: 1.374491 [12800/60000]\n", - "loss: 1.446650 [19200/60000]\n", - "loss: 1.322406 [25600/60000]\n", - "loss: 1.325748 [32000/60000]\n", - "loss: 1.331345 [38400/60000]\n", - "loss: 1.261354 [44800/60000]\n", - "loss: 1.292640 [51200/60000]\n", - "loss: 1.200176 [57600/60000]\n", - "Test Error: \n", - " Accuracy: 63.3%, Avg loss: 1.229079 \n", - "\n", - "Epoch 5\n", - "-------------------------------\n", - "loss: 1.302291 [ 0/60000]\n", - "loss: 1.292532 [ 6400/60000]\n", - "loss: 1.125565 [12800/60000]\n", - "loss: 1.234343 [19200/60000]\n", - "loss: 1.108794 [25600/60000]\n", - "loss: 1.134641 [32000/60000]\n", - "loss: 1.149444 [38400/60000]\n", - "loss: 1.086560 [44800/60000]\n", - "loss: 1.125073 [51200/60000]\n", - "loss: 1.049052 [57600/60000]\n", - "Test Error: \n", - " Accuracy: 64.7%, Avg loss: 1.072207 \n", - "\n", - "Done!\n" - ] - } - ], - "source": [ - "epochs = 5\n", - "for t in range(epochs):\n", - " print(f\"Epoch {t+1}\\n-------------------------------\")\n", - " train(train_dataloader, model, loss_fn, optimizer)\n", - " test(test_dataloader, model, loss_fn)\n", - "print(\"Done!\")" - ] - }, - { - "cell_type": "markdown", - "id": "85d97839", - "metadata": {}, - "source": [ - "### Save Model State Dict\n", - "This is the [currently recommended save format](https://pytorch.org/tutorials/beginner/saving_loading_models.html#saving-loading-model-for-inference)." - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "5d5d24de", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Saved PyTorch Model State to model_weights.pt\n" - ] - } - ], - "source": [ - "torch.save(model.state_dict(), \"model_weights.pt\")\n", - "print(\"Saved PyTorch Model State to model_weights.pt\")" - ] - }, - { - "cell_type": "markdown", - "id": "7416bbef", - "metadata": {}, - "source": [ - "### Save Entire Model\n", - "This saves the entire model using python pickle, but has the [following disadvantage](https://pytorch.org/tutorials/beginner/saving_loading_models.html#save-load-entire-model):\n", - "> The serialized data is bound to the specific classes and the exact directory structure used when the model is saved... Because of this, your code can break in various ways when used in other projects or after refactors." - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "e87098c0", - "metadata": {}, - "outputs": [], - "source": [ - "torch.save(model, \"model.pt\")" - ] - }, - { - "cell_type": "markdown", - "id": "ac221ca7-e227-4c8c-8577-1eeda4a61fc7", - "metadata": {}, - "source": [ - "### Save Model as TorchScript\n", - "This saves an [intermediate representation of the compute graph](https://pytorch.org/tutorials/beginner/saving_loading_models.html#export-load-model-in-torchscript-format), which does not require pickle (or even python). However, this currently doesn't work with spark, which uses pickle serialization." - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "6d9b3a45-7618-43e4-8bd3-8bb317a484d3", - "metadata": {}, - "outputs": [], - "source": [ - "scripted = torch.jit.script(model)" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "0cd01550-c72e-47f2-abe6-e14f26b06fc7", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "scripted.save(\"model.ts\")" - ] - }, - { - "cell_type": "markdown", - "id": "12ee8916-f437-4a2a-9bf4-14ff5376d305", - "metadata": {}, - "source": [ - "### Load Model State" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "8fe3b5d1", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "model_from_state = NeuralNetwork()\n", - "model_from_state.load_state_dict(torch.load(\"model_weights.pt\"))" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "0c405bd0", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Predicted: \"Ankle boot\", Actual: \"Ankle boot\"\n" - ] - } - ], - "source": [ - "model_from_state.eval()\n", - "x, y = test_data[0][0], test_data[0][1]\n", - "with torch.no_grad():\n", - " pred = model_from_state(x)\n", - " predicted, actual = classes[pred[0].argmax(0)], classes[y]\n", - " print(f'Predicted: \"{predicted}\", Actual: \"{actual}\"')" - ] - }, - { - "cell_type": "markdown", - "id": "1708f5e0", - "metadata": {}, - "source": [ - "### Load Model" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "dc5bce69", - "metadata": {}, - "outputs": [], - "source": [ - "new_model = torch.load(\"model.pt\")" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "cc219a56-4abd-4b61-9f9a-686dae7c9614", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Predicted: \"Ankle boot\", Actual: \"Ankle boot\"\n" - ] - } - ], - "source": [ - "x, y = test_data[0][0], test_data[0][1]\n", - "with torch.no_grad():\n", - " pred = new_model(x)\n", - " predicted, actual = classes[pred[0].argmax(0)], classes[y]\n", - " print(f'Predicted: \"{predicted}\", Actual: \"{actual}\"')" - ] - }, - { - "cell_type": "markdown", - "id": "290c482a-1c5d-4bf2-bc3f-8a4e53d442b5", - "metadata": {}, - "source": [ - "### Load Torchscript Model" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "ef3c419e-d384-446c-b07b-1af93e07d6c0", - "metadata": {}, - "outputs": [], - "source": [ - "ts_model = torch.jit.load(\"model.ts\")" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "id": "038af830-a360-45eb-ab4e-b1adab0af164", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Predicted: \"Ankle boot\", Actual: \"Ankle boot\"\n" - ] - } - ], - "source": [ - "x, y = test_data[0][0], test_data[0][1]\n", - "with torch.no_grad():\n", - " pred = ts_model(x)\n", - " predicted, actual = classes[pred[0].argmax(0)], classes[y]\n", - " print(f'Predicted: \"{predicted}\", Actual: \"{actual}\"')" - ] - }, - { - "cell_type": "markdown", - "id": "ad918393", - "metadata": {}, - "source": [ - "## PySpark" - ] - }, - { - "cell_type": "markdown", - "id": "fd1daec3", - "metadata": {}, - "source": [ - "### Convert numpy dataset to Pandas DataFrame" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "id": "42c5feba", - "metadata": {}, - "outputs": [], - "source": [ - "import pandas as pd\n", - "from pyspark.sql.types import StructType, StructField, ArrayType, FloatType" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "id": "f063cbe7", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "((10000, 28, 28), dtype('uint8'))" - ] - }, - "execution_count": 22, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "data = test_data.data.numpy()\n", - "data.shape, data.dtype" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "id": "8c828393", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "((10000, 784), dtype('float64'))" - ] - }, - "execution_count": 23, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "data = data.reshape(10000, 784) / 255.0\n", - "data.shape, data.dtype" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "id": "7760bdbe", - "metadata": {}, - "outputs": [ - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
0123456789...774775776777778779780781782783
00.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
10.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0078430.0117650.00.0117650.6823530.7411760.2627450.00.00.0
20.00.00.00.0000000.00.0000000.0000000.00.0039220.000000...0.6431370.2274510.00.0000000.0000000.0000000.0000000.00.00.0
30.00.00.00.0000000.00.0000000.0000000.00.0000000.082353...0.0039220.0000000.00.0000000.0000000.0000000.0000000.00.00.0
40.00.00.00.0078430.00.0039220.0039220.00.0000000.000000...0.2784310.0470590.00.0000000.0000000.0000000.0000000.00.00.0
..................................................................
99950.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
99960.00.00.00.0000000.00.0000000.0000000.00.0000000.121569...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
99970.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.1058820.0000000.00.0000000.0000000.0000000.0000000.00.00.0
99980.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
99990.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
\n", - "

10000 rows × 784 columns

\n", - "
" - ], - "text/plain": [ - " 0 1 2 3 4 5 6 7 8 \n", - "0 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \\\n", - "1 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", - "2 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.003922 \n", - "3 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", - "4 0.0 0.0 0.0 0.007843 0.0 0.003922 0.003922 0.0 0.000000 \n", - "... ... ... ... ... ... ... ... ... ... \n", - "9995 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", - "9996 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", - "9997 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", - "9998 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", - "9999 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", - "\n", - " 9 ... 774 775 776 777 778 779 \n", - "0 0.000000 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \\\n", - "1 0.000000 ... 0.007843 0.011765 0.0 0.011765 0.682353 0.741176 \n", - "2 0.000000 ... 0.643137 0.227451 0.0 0.000000 0.000000 0.000000 \n", - "3 0.082353 ... 0.003922 0.000000 0.0 0.000000 0.000000 0.000000 \n", - "4 0.000000 ... 0.278431 0.047059 0.0 0.000000 0.000000 0.000000 \n", - "... ... ... ... ... ... ... ... ... \n", - "9995 0.000000 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \n", - "9996 0.121569 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \n", - "9997 0.000000 ... 0.105882 0.000000 0.0 0.000000 0.000000 0.000000 \n", - "9998 0.000000 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \n", - "9999 0.000000 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \n", - "\n", - " 780 781 782 783 \n", - "0 0.000000 0.0 0.0 0.0 \n", - "1 0.262745 0.0 0.0 0.0 \n", - "2 0.000000 0.0 0.0 0.0 \n", - "3 0.000000 0.0 0.0 0.0 \n", - "4 0.000000 0.0 0.0 0.0 \n", - "... ... ... ... ... \n", - "9995 0.000000 0.0 0.0 0.0 \n", - "9996 0.000000 0.0 0.0 0.0 \n", - "9997 0.000000 0.0 0.0 0.0 \n", - "9998 0.000000 0.0 0.0 0.0 \n", - "9999 0.000000 0.0 0.0 0.0 \n", - "\n", - "[10000 rows x 784 columns]" - ] - }, - "execution_count": 24, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "pdf784 = pd.DataFrame(data)\n", - "pdf784" - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "id": "f7d2bc0d", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 178 ms, sys: 52.5 ms, total: 231 ms\n", - "Wall time: 227 ms\n" - ] - }, - { - "data": { - "text/html": [ - "
\n", - "\n", - "\n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - " \n", - "
data
0[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
1[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
2[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.003...
3[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
4[0.0, 0.0, 0.0, 0.00784313725490196, 0.0, 0.00...
......
9995[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
9996[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
9997[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
9998[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
9999[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
\n", - "

10000 rows × 1 columns

\n", - "
" - ], - "text/plain": [ - " data\n", - "0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - "1 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - "2 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.003...\n", - "3 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - "4 [0.0, 0.0, 0.0, 0.00784313725490196, 0.0, 0.00...\n", - "... ...\n", - "9995 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - "9996 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - "9997 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - "9998 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - "9999 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - "\n", - "[10000 rows x 1 columns]" - ] - }, - "execution_count": 25, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "%%time\n", - "# 1 column of array\n", - "pdf1 = pd.DataFrame()\n", - "pdf1['data'] = pdf784.values.tolist()\n", - "pdf1" - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "id": "20c3b86a-8b61-4128-ab61-0199fd3437cc", - "metadata": {}, - "outputs": [], - "source": [ - "### Create Spark DataFrame from Pandas DataFrame" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "id": "4863d5ff", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 3.33 s, sys: 68.3 ms, total: 3.4 s\n", - "Wall time: 5.51 s\n" - ] - } - ], - "source": [ - "%%time\n", - "# force FloatType since Spark defaults to DoubleType\n", - "schema = StructType([StructField(\"data\",ArrayType(FloatType()), True)])\n", - "df = spark.createDataFrame(pdf1, schema)" - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "id": "406edba5", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "StructType([StructField('data', ArrayType(FloatType(), True), True)])" - ] - }, - "execution_count": 28, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "df.schema" - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "id": "831f4a01-3a49-4114-b9a0-2ae54526d72d", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 4.35 s, sys: 141 ms, total: 4.5 s\n", - "Wall time: 4.75 s\n" - ] - } - ], - "source": [ - "%%time\n", - "# force FloatType since Spark defaults to DoubleType\n", - "schema = StructType([StructField(str(x), FloatType(), True) for x in range(784)])\n", - "df784 = spark.createDataFrame(pdf784, schema)" - ] - }, - { - "cell_type": "markdown", - "id": "ac4c7448", - "metadata": {}, - "source": [ - "### Save the test dataset as parquet files" - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "id": "e8ebae46", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "23/05/19 17:55:11 WARN TaskSetManager: Stage 0 contains a task of very large size (7070 KiB). The maximum recommended task size is 1000 KiB.\n", - "[Stage 0:> (0 + 8) / 8]\r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 5.9 ms, sys: 4.12 ms, total: 10 ms\n", - "Wall time: 4.31 s\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - } - ], - "source": [ - "%%time\n", - "df.write.mode(\"overwrite\").parquet(\"fashion_mnist_1\")" - ] - }, - { - "cell_type": "code", - "execution_count": 31, - "id": "922314ce-2996-4666-9fc9-bcd98d16bb56", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "23/05/19 17:55:14 WARN package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.\n", - "23/05/19 17:55:14 WARN TaskSetManager: Stage 1 contains a task of very large size (7067 KiB). The maximum recommended task size is 1000 KiB.\n", - "[Stage 1:> (0 + 8) / 8]\r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 11.9 ms, sys: 54 µs, total: 12 ms\n", - "Wall time: 2.93 s\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - } - ], - "source": [ - "%%time\n", - "df784.write.mode(\"overwrite\").parquet(\"fashion_mnist_784\")" - ] - }, - { - "cell_type": "markdown", - "id": "8688429e", - "metadata": {}, - "source": [ - "### Check arrow memory configuration" - ] - }, - { - "cell_type": "code", - "execution_count": 32, - "id": "088cb37f", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "23/05/19 17:55:17 WARN TaskSetManager: Stage 2 contains a task of very large size (7070 KiB). The maximum recommended task size is 1000 KiB.\n" - ] - } - ], - "source": [ - "spark.conf.set(\"spark.sql.execution.arrow.maxRecordsPerBatch\", \"128\")\n", - "# This line will fail if the vectorized reader runs out of memory\n", - "assert len(df.head()) > 0, \"`df` should not be empty\"" - ] - }, - { - "cell_type": "markdown", - "id": "d7c77eb4-7bd6-40c7-9a35-ee899a66ece3", - "metadata": {}, - "source": [ - "## Inference using Spark DL API" - ] - }, - { - "cell_type": "markdown", - "id": "59395856-a588-43c6-93c8-c83100716ac1", - "metadata": { - "tags": [] - }, - "source": [ - "### 1 columns of 784 float" - ] - }, - { - "cell_type": "code", - "execution_count": 33, - "id": "133cc9a5-64c6-4820-807e-b87cf7e0b75a", - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "\n", - "from pyspark.ml.functions import predict_batch_udf\n", - "from pyspark.sql.functions import struct, col, array\n", - "from pyspark.sql.types import ArrayType, FloatType, Union, Dict" - ] - }, - { - "cell_type": "code", - "execution_count": 34, - "id": "79b151d9-d112-43b6-a479-887e2fd0e2b1", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "1" - ] - }, - "execution_count": 34, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "df = spark.read.parquet(\"fashion_mnist_1\")\n", - "len(df.columns)" - ] - }, - { - "cell_type": "code", - "execution_count": 35, - "id": "cabcd546-2e8e-40d0-8b79-7598a7a83aae", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "StructType([StructField('data', ArrayType(FloatType(), True), True)])" - ] - }, - "execution_count": 35, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "df.schema" - ] - }, - { - "cell_type": "code", - "execution_count": 36, - "id": "5db4b957-57fc-4bc5-b8bb-db0657a186c8", - "metadata": {}, - "outputs": [], - "source": [ - "# get absolute path to model\n", - "model_dir = \"{}/model.ts\".format(os.getcwd())" - ] - }, - { - "cell_type": "code", - "execution_count": 37, - "id": "73dc73cb-25e3-4798-a019-e1abd684eaa1", - "metadata": {}, - "outputs": [], - "source": [ - "def predict_batch_fn():\n", - " import torch\n", - " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", - " print(\"Using {} device\".format(device))\n", - " model = torch.jit.load(model_dir)\n", - " model.to(device)\n", - " \n", - " def predict(inputs: np.ndarray):\n", - " torch_inputs = torch.from_numpy(inputs).to(device)\n", - " outputs = model(torch_inputs)\n", - " return outputs.detach().numpy()\n", - " \n", - " return predict" - ] - }, - { - "cell_type": "code", - "execution_count": 38, - "id": "df68cca1-2d47-4e88-8aad-9899402aee97", - "metadata": {}, - "outputs": [], - "source": [ - "mnist = predict_batch_udf(predict_batch_fn,\n", - " input_tensor_shapes=[[784]],\n", - " return_type=ArrayType(FloatType()),\n", - " batch_size=50)" - ] - }, - { - "cell_type": "code", - "execution_count": 39, - "id": "63555b3b-3673-4712-97aa-fd728c6c4979", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 465 ms, sys: 117 ms, total: 582 ms\n", - "Wall time: 4.31 s\n" - ] - } - ], - "source": [ - "%%time\n", - "# first pass caches model/fn\n", - "preds = df.withColumn(\"preds\", mnist(struct(df.columns))).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 40, - "id": "5dbf058a-70d6-4199-af9d-13843d078950", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 449 ms, sys: 125 ms, total: 574 ms\n", - "Wall time: 1.48 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", mnist(*df.columns)).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 41, - "id": "3f5ed801-6ca5-43a0-bf9c-2535a0dfe2e8", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 698 ms, sys: 117 ms, total: 815 ms\n", - "Wall time: 1.52 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", mnist(*[col(c) for c in df.columns])).collect()" - ] - }, - { - "cell_type": "markdown", - "id": "c6dbec03-9b64-46c4-a748-f889be571384", - "metadata": { - "tags": [] - }, - "source": [ - "### Check predictions" - ] - }, - { - "cell_type": "code", - "execution_count": 42, - "id": "f1f1e5fd-5866-4b78-b9d3-709e6b383a0c", - "metadata": {}, - "outputs": [], - "source": [ - "predictions = preds[0].preds\n", - "img = preds[0].data" - ] - }, - { - "cell_type": "code", - "execution_count": 43, - "id": "76b76502-adb7-45ec-a365-2e61cdd576fc", - "metadata": {}, - "outputs": [], - "source": [ - "import matplotlib.pyplot as plt\n", - "import numpy as np" - ] - }, - { - "cell_type": "code", - "execution_count": 44, - "id": "c163953a-1504-444f-b39f-86b61d34e440", - "metadata": {}, - "outputs": [], - "source": [ - "img = np.array(img).reshape(28,28)" - ] - }, - { - "cell_type": "code", - "execution_count": 45, - "id": "bc0fad05-50ab-4ae5-b9fd-e50133c4c92a", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAfwElEQVR4nO3df3DU9b3v8dfm10Iw2RBCfkmgARGsSGyppFSlWHKBOMcB5d7xV88FjwevNDhFtHrpqGjbmbR4xzp6qN6Z24KeEbXeERi9LR0EE8YasKAcSrUpSVOBkgTFZjck5ud+7h8c064Q4LNueCfh+ZjZGbL7feX75ptv8so3u/kk4JxzAgDgPEuyHgAAcGGigAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGAixXqAz4tGozp69KgyMjIUCASsxwEAeHLOqbW1VYWFhUpK6v86Z9AV0NGjR1VUVGQ9BgDgCzp8+LDGjRvX7+ODroAyMjIkSdfoeqUo1XgaAICvHnXrLf2q7+t5fwasgNatW6fHH39cTU1NKikp0dNPP62ZM2eeNffZj91SlKqUAAUEAEPOf64weranUQbkRQgvv/yyVq1apTVr1ujdd99VSUmJ5s+fr2PHjg3E7gAAQ9CAFNATTzyhZcuW6Y477tCXv/xlPfvss0pPT9cvfvGLgdgdAGAISngBdXV1ae/evSorK/v7TpKSVFZWppqamlO27+zsVCQSibkBAIa/hBfQxx9/rN7eXuXl5cXcn5eXp6amplO2r6ysVCgU6rvxCjgAuDCY/yLq6tWrFQ6H+26HDx+2HgkAcB4k/FVwOTk5Sk5OVnNzc8z9zc3Nys/PP2X7YDCoYDCY6DEAAINcwq+A0tLSNGPGDG3fvr3vvmg0qu3bt2vWrFmJ3h0AYIgakN8DWrVqlZYsWaKvfe1rmjlzpp588km1tbXpjjvuGIjdAQCGoAEpoJtvvlkfffSRHnnkETU1NenKK6/U1q1bT3lhAgDgwhVwzjnrIf5RJBJRKBTSHC1kJQQAGIJ6XLeqtEXhcFiZmZn9bmf+KjgAwIWJAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJhJeQI8++qgCgUDMberUqYneDQBgiEsZiHd6+eWX64033vj7TlIGZDcAgCFsQJohJSVF+fn5A/GuAQDDxIA8B3Tw4EEVFhZq4sSJuv3223Xo0KF+t+3s7FQkEom5AQCGv4QXUGlpqTZs2KCtW7fqmWeeUUNDg6699lq1traedvvKykqFQqG+W1FRUaJHAgAMQgHnnBvIHbS0tGjChAl64okndOedd57yeGdnpzo7O/vejkQiKioq0hwtVEogdSBHAwAMgB7XrSptUTgcVmZmZr/bDfirA7KysnTppZeqrq7utI8Hg0EFg8GBHgMAMMgM+O8BnThxQvX19SooKBjoXQEAhpCEF9D999+v6upq/eUvf9Hbb7+tG2+8UcnJybr11lsTvSsAwBCW8B/BHTlyRLfeequOHz+usWPH6pprrtGuXbs0duzYRO8KADCEJbyAXnrppUS/SwDAMMRacAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAEwM+B+kw38KBPwzcfyx2kCK/4fU9fR4Z07u7Pz8n4aj5NGjvTN/fHSKd2bStL96ZyQp/Pw478zo52ri2tdwkzRqlHcmns9B9w9/SXqo4goIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCC1bDPl/O0CnQ8q+rGs4J23JKTvSOuq8s7E4hjP9IXWBncU+OGPO9MoN5/P+V5f/APSZr28FbvzL/9y1zvTPP/KfbOZBzxXwU6+c13vTOSlDx5onemc7z/SueH56Z5Zyav+9A7I0k9fz3qH/Je+T4gncOXPK6AAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmGAx0uHGe9HA87cApyTpPO3rfP6fjj7wDe/MXZf8P+/ME03zvDNP1/gvECpJz1z3vHfmXwt3emd2rPzYO7P72ATvTPt0/4+RJI1qjnpn2vL8v6+/ruw978xbrV/xzkjSuEr/xUh9F/cNuKh0Dp+CXAEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwwWKkw41z1hOcUdf8r3ln0v/0kXemp+FD70y88n7X4Z1p/+c070zq3/w/XaN+a0j2ue8//qt35s4pNd6ZJwv2eGeK9/gvwjm2Jb7Pi5ZL/L9H75l2Iq59+fp0XO952c9A4goIAGCCAgIAmPAuoJ07d+qGG25QYWGhAoGANm/eHPO4c06PPPKICgoKNHLkSJWVlengwYOJmhcAMEx4F1BbW5tKSkq0bt260z6+du1aPfXUU3r22We1e/dujRo1SvPnz1dHh//PyQEAw5f3s5rl5eUqLy8/7WPOOT355JN66KGHtHDhQknS888/r7y8PG3evFm33HLLF5sWADBsJPQ5oIaGBjU1NamsrKzvvlAopNLSUtXUnP4VMp2dnYpEIjE3AMDwl9ACampqkiTl5eXF3J+Xl9f32OdVVlYqFAr13YqKihI5EgBgkDJ/Fdzq1asVDof7bocPH7YeCQBwHiS0gPLz8yVJzc3NMfc3Nzf3PfZ5wWBQmZmZMTcAwPCX0AIqLi5Wfn6+tm/f3ndfJBLR7t27NWvWrETuCgAwxHm/Cu7EiROqq6vre7uhoUH79u1Tdna2xo8fr5UrV+pHP/qRJk+erOLiYj388MMqLCzUokWLEjk3AGCI8y6gPXv26Lrrrut7e9WqVZKkJUuWaMOGDXrggQfU1tamu+66Sy0tLbrmmmu0detWjRgxInFTAwCGvIBzg2v1ykgkolAopDlaqJRA6sDuLBCILze4DtkXduT734grd1n5n7wzew9+yTtzU8m73pmt//fr3hlJGlf5dlw5X3/6uf+irFdfVnf2jT7nnarLvDOS1DvC/xy/9Hn/X6GovcP/Od/Rf/D/vI2mxvm5Pu8T70jbp/4LzfZ0+H+tm7x0r3cmbp5fK3tct6rcZoXD4TM+r2/+KjgAwIWJAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGDC+88xDFpxrGwdSE6Oa1eupyeunK+OG2Z6Zzb82xPemW3tx7wzklT59vXemaSw/ym36Q9Xemfu++fXvDOS9L/Gl3tnLl3+jn/mzj3emQ9eu9Q7kzrFf4VqSYo2ZHhnjqzx30/eK/6ZyAT/z/VR13zkvyNJHx3N8s4snflb78zuW6d5Z3q9E4MPV0AAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMBJxzznqIfxSJRBQKhTRHC5USSLUex9S3ft/mnfnlX77inQnXZntnJCm/xv/U6fqXT7wzbb8d650ZMetj74wkjUrr9s50P5fnncncuMs7E4+mzZedl/1IUuSTUd6ZoouPe2eOfpzlnUk6PMI7I0nBKWHvTPqrIe9M1r/XeGfiWYBZkhTPl3zPffW4blW5zQqHw8rMzOx3O66AAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmEixHsCSm1USVy6pJ+qd6bnIf2HV/13ln3Hpvd6ZlMIO74wkHb/Ff19zcw95Z7aNyvHOfNLkvyCkJEVzW70zXV/y/z6u/+UZEyt/0Qdx5T785RXemeyxEe/M4UP+H9v07HbvzLirPvLOSNLB9y/2zhTGsbBo8hkW7OxPtN3/OEiS6+mJI+S5gOk5bs8VEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABODdjHS6DXTFU0Zcc7btxYFvfcxqrHbOyNJ0ajnwnySjn3l3P8vn/mfZZu9Mz/+3QLvTHp6p3dGki4b2+yd2f9JoXemO9N/8df0P/sv5CpJSb8b7Z0Z2et/Ppz4b6Xema6L/L9fTDvhf+wkafxP/ReoPbzK/3PwkolN3pk/N/ovYDru4hbvjCQ11Y33zjTf8w3vTOoJ/3MoGInvYxuPzG1+i9o61yWdw9q0XAEBAExQQAAAE94FtHPnTt1www0qLCxUIBDQ5s2bYx5funSpAoFAzG3BAv8fCwEAhjfvAmpra1NJSYnWrVvX7zYLFixQY2Nj3+3FF1/8QkMCAIYf7xchlJeXq7y8/IzbBINB5efnxz0UAGD4G5DngKqqqpSbm6spU6Zo+fLlOn78eL/bdnZ2KhKJxNwAAMNfwgtowYIFev7557V9+3b95Cc/UXV1tcrLy9Xb23va7SsrKxUKhfpuRUVFiR4JADAIJfz3gG655Za+f19xxRWaPn26Jk2apKqqKs2dO/eU7VevXq1Vq1b1vR2JRCghALgADPjLsCdOnKicnBzV1dWd9vFgMKjMzMyYGwBg+BvwAjpy5IiOHz+ugoKCgd4VAGAI8f4R3IkTJ2KuZhoaGrRv3z5lZ2crOztbjz32mBYvXqz8/HzV19frgQce0CWXXKL58+cndHAAwNDmXUB79uzRdddd1/f2Z8/fLFmyRM8884z279+v5557Ti0tLSosLNS8efP0wx/+UMGg/zpRAIDhy7uA5syZI+f6XzjvN7/5zRca6DN/XpyqpJHnvqDkxW/4L+aXsuNd74wkdV7/Ne9MT7r/fhq7srwz11/2B+/Mx12jvDOS9B9HL/bOdB7131dyR8A70z6+xzsjSWPe98+0j032zoT+8DfvzInJWd6ZeB1a4P9xKsj6q3fmk/aR3pmLx7Z4Z/Y0xffCpouOnv7Vu2cSTfU/X7Pe9//1k8CnXd4ZSYpm+h/zhpXTvLbv7eyQfnz27VgLDgBgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgIuF/kjtRxu5OVnLaua8yPPl7B7z3cezd8d4ZSUr+03HvTOol+d6ZvS3+8/3+A/9M6if+qzlLUk8oGlfOV29Bp3cme/SJuPZ1/PIc78zFb7Z5Z6Kj/P88yUVvN3hn1Ol/7CTpyLe+7J1JS/JfOToeuemt3pmW38T3BzEzD7Z4ZwInPvXO9GZf5J1J9t+NJOnPi/33FU32+2sD0dRz254rIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYG7WKkoZfeUUog9Zy33zn/K977+Pkbz3lnJOm+n/wP70wgjnU7P2of5Z3JzPdfqDF9Qpd3RpJ6o/7fv3zS4r8QYm+7/2kajWM2SZrwX/7inakfNcE7M+b3fos7StLxRZd4Z8Z//Yh3RpKuTK33zvQ4/0Vts9PjXFHTU9b8xrhytVeO9s5Mm/CJd6Y5cu5f6z7T2pbunZGkqQX+i9p+8pTfOd7THdWH57AdV0AAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMBJxz/qsiDqBIJKJQKKQ5Wui1GGk8Om6YGVcu8q8R70xBpn/mvxfWeGfqOvO8Mx3R+I7ziKRu70wo2X/xyfqOsd6Zpo5M70y8jnf4Lxpb/8dC78yIRv/FPgtqOr0zkhQ89Df/UKr/orE9oZHemUCv/8q+nWNGeGckKZoW8M6kfOo/X1K3f6Yn3f98kKTudP/rjsxfHfDavsd1aUfbiwqHw8rM7P9zkSsgAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJvxXDxxGRrz2Tpw5/4ybNtU7s2HkP3lnOnL8F13siWNxQim+RQ1TOv3Xvk1t7fXOjGxs885IUtLfWv0zHx72zkyWf+Z88j/i8UlKieNLULL/IpzBri7//UhKzsiIK+cr2t7unUkbPTquffV+9JF3xnep1Kg7t4WKuQICAJiggAAAJrwKqLKyUldddZUyMjKUm5urRYsWqba2Nmabjo4OVVRUaMyYMbrooou0ePFiNTc3J3RoAMDQ51VA1dXVqqio0K5du7Rt2zZ1d3dr3rx5amv7+8/b7733Xr322mt65ZVXVF1draNHj+qmm25K+OAAgKHN6xnArVu3xry9YcMG5ebmau/evZo9e7bC4bB+/vOfa+PGjfrWt74lSVq/fr0uu+wy7dq1S1//+tcTNzkAYEj7Qs8BhcNhSVJ2drYkae/everu7lZZWVnfNlOnTtX48eNVU3P6Py/d2dmpSCQScwMADH9xF1A0GtXKlSt19dVXa9q0aZKkpqYmpaWlKSsrK2bbvLw8NTU1nfb9VFZWKhQK9d2KioriHQkAMITEXUAVFRU6cOCAXnrppS80wOrVqxUOh/tuhw8P7t+PAAAkRly/iLpixQq9/vrr2rlzp8aNG9d3f35+vrq6utTS0hJzFdTc3Kz8/PzTvq9gMKhgMBjPGACAIczrCsg5pxUrVmjTpk3asWOHiouLYx6fMWOGUlNTtX379r77amtrdejQIc2aNSsxEwMAhgWvK6CKigpt3LhRW7ZsUUZGRt/zOqFQSCNHjlQoFNKdd96pVatWKTs7W5mZmbrnnns0a9YsXgEHAIjhVUDPPPOMJGnOnDkx969fv15Lly6VJP30pz9VUlKSFi9erM7OTs2fP18/+9nPEjIsAGD4CDjn/FeHHECRSEShUEhztFApgVTrcQAAnnpct6q0ReFwWJmZmf1ux1pwAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMOFVQJWVlbrqqquUkZGh3NxcLVq0SLW1tTHbzJkzR4FAIOZ29913J3RoAMDQ51VA1dXVqqio0K5du7Rt2zZ1d3dr3rx5amtri9lu2bJlamxs7LutXbs2oUMDAIa+FJ+Nt27dGvP2hg0blJubq71792r27Nl996enpys/Pz8xEwIAhqUv9BxQOByWJGVnZ8fc/8ILLygnJ0fTpk3T6tWr1d7e3u/76OzsVCQSibkBAIY/ryugfxSNRrVy5UpdffXVmjZtWt/9t912myZMmKDCwkLt379fDz74oGpra/Xqq6+e9v1UVlbqsccei3cMAMAQFXDOuXiCy5cv169//Wu99dZbGjduXL/b7dixQ3PnzlVdXZ0mTZp0yuOdnZ3q7OzsezsSiaioqEhztFApgdR4RgMAGOpx3arSFoXDYWVmZva7XVxXQCtWrNDrr7+unTt3nrF8JKm0tFSS+i2gYDCoYDAYzxgAgCHMq4Ccc7rnnnu0adMmVVVVqbi4+KyZffv2SZIKCgriGhAAMDx5FVBFRYU2btyoLVu2KCMjQ01NTZKkUCikkSNHqr6+Xhs3btT111+vMWPGaP/+/br33ns1e/ZsTZ8+fUD+AwCAocnrOaBAIHDa+9evX6+lS5fq8OHD+va3v60DBw6ora1NRUVFuvHGG/XQQw+d8eeA/ygSiSgUCvEcEAAMUQPyHNDZuqqoqEjV1dU+7xIAcIFiLTgAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgIkU6wE+zzknSepRt+SMhwEAeOtRt6S/fz3vz6AroNbWVknSW/qV8SQAgC+itbVVoVCo38cD7mwVdZ5Fo1EdPXpUGRkZCgQCMY9FIhEVFRXp8OHDyszMNJrQHsfhJI7DSRyHkzgOJw2G4+CcU2trqwoLC5WU1P8zPYPuCigpKUnjxo074zaZmZkX9An2GY7DSRyHkzgOJ3EcTrI+Dme68vkML0IAAJiggAAAJoZUAQWDQa1Zs0bBYNB6FFMch5M4DidxHE7iOJw0lI7DoHsRAgDgwjCkroAAAMMHBQQAMEEBAQBMUEAAABNDpoDWrVunL33pSxoxYoRKS0v1zjvvWI903j366KMKBAIxt6lTp1qPNeB27typG264QYWFhQoEAtq8eXPM4845PfLIIyooKNDIkSNVVlamgwcP2gw7gM52HJYuXXrK+bFgwQKbYQdIZWWlrrrqKmVkZCg3N1eLFi1SbW1tzDYdHR2qqKjQmDFjdNFFF2nx4sVqbm42mnhgnMtxmDNnzinnw91332008ekNiQJ6+eWXtWrVKq1Zs0bvvvuuSkpKNH/+fB07dsx6tPPu8ssvV2NjY9/trbfesh5pwLW1tamkpETr1q077eNr167VU089pWeffVa7d+/WqFGjNH/+fHV0dJznSQfW2Y6DJC1YsCDm/HjxxRfP44QDr7q6WhUVFdq1a5e2bdum7u5uzZs3T21tbX3b3HvvvXrttdf0yiuvqLq6WkePHtVNN91kOHXinctxkKRly5bFnA9r1641mrgfbgiYOXOmq6io6Hu7t7fXFRYWusrKSsOpzr81a9a4kpIS6zFMSXKbNm3qezsajbr8/Hz3+OOP993X0tLigsGge/HFFw0mPD8+fxycc27JkiVu4cKFJvNYOXbsmJPkqqurnXMnP/apqanulVde6dvmgw8+cJJcTU2N1ZgD7vPHwTnnvvnNb7rvfve7dkOdg0F/BdTV1aW9e/eqrKys776kpCSVlZWppqbGcDIbBw8eVGFhoSZOnKjbb79dhw4dsh7JVENDg5qammLOj1AopNLS0gvy/KiqqlJubq6mTJmi5cuX6/jx49YjDahwOCxJys7OliTt3btX3d3dMefD1KlTNX78+GF9Pnz+OHzmhRdeUE5OjqZNm6bVq1ervb3dYrx+DbrFSD/v448/Vm9vr/Ly8mLuz8vL0x//+EejqWyUlpZqw4YNmjJlihobG/XYY4/p2muv1YEDB5SRkWE9nommpiZJOu358dljF4oFCxbopptuUnFxserr6/X9739f5eXlqqmpUXJysvV4CReNRrVy5UpdffXVmjZtmqST50NaWpqysrJith3O58PpjoMk3XbbbZowYYIKCwu1f/9+Pfjgg6qtrdWrr75qOG2sQV9A+Lvy8vK+f0+fPl2lpaWaMGGCfvnLX+rOO+80nAyDwS233NL37yuuuELTp0/XpEmTVFVVpblz5xpONjAqKip04MCBC+J50DPp7zjcddddff++4oorVFBQoLlz56q+vl6TJk0632Oe1qD/EVxOTo6Sk5NPeRVLc3Oz8vPzjaYaHLKysnTppZeqrq7OehQzn50DnB+nmjhxonJycobl+bFixQq9/vrrevPNN2P+fEt+fr66urrU0tISs/1wPR/6Ow6nU1paKkmD6nwY9AWUlpamGTNmaPv27X33RaNRbd++XbNmzTKczN6JEydUX1+vgoIC61HMFBcXKz8/P+b8iEQi2r179wV/fhw5ckTHjx8fVueHc04rVqzQpk2btGPHDhUXF8c8PmPGDKWmpsacD7W1tTp06NCwOh/OdhxOZ9++fZI0uM4H61dBnIuXXnrJBYNBt2HDBvf++++7u+66y2VlZbmmpibr0c6r++67z1VVVbmGhgb329/+1pWVlbmcnBx37Ngx69EGVGtrq3vvvffce++95yS5J554wr333nvuww8/dM459+Mf/9hlZWW5LVu2uP3797uFCxe64uJi9+mnnxpPnlhnOg6tra3u/vvvdzU1Na6hocG98cYb7qtf/aqbPHmy6+josB49YZYvX+5CoZCrqqpyjY2Nfbf29va+be6++243fvx4t2PHDrdnzx43a9YsN2vWLMOpE+9sx6Gurs794Ac/cHv27HENDQ1uy5YtbuLEiW727NnGk8caEgXknHNPP/20Gz9+vEtLS3MzZ850u3btsh7pvLv55ptdQUGBS0tLcxdffLG7+eabXV1dnfVYA+7NN990kk65LVmyxDl38qXYDz/8sMvLy3PBYNDNnTvX1dbW2g49AM50HNrb2928efPc2LFjXWpqqpswYYJbtmzZsPsm7XT/f0lu/fr1fdt8+umn7jvf+Y4bPXq0S09PdzfeeKNrbGy0G3oAnO04HDp0yM2ePdtlZ2e7YDDoLrnkEve9733PhcNh28E/hz/HAAAwMeifAwIADE8UEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBM/H+wVOxLQwVbkwAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "plt.figure()\n", - "plt.imshow(img)\n", - "plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 46, - "id": "56f36efb-e3a2-49f9-b9fb-1657bc25e5c5", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[-1.7200822830200195,\n", - " -2.013300895690918,\n", - " -0.3198302984237671,\n", - " -1.4132847785949707,\n", - " -0.25933510065078735,\n", - " 1.7492953538894653,\n", - " -0.45816969871520996,\n", - " 2.1313018798828125,\n", - " 1.6222647428512573,\n", - " 1.3297781944274902]" - ] - }, - "execution_count": 46, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "predictions" - ] - }, - { - "cell_type": "markdown", - "id": "56ca1195-ea0f-405f-87fe-857e5c0c76a5", - "metadata": {}, - "source": [ - "### 784 columns of float" - ] - }, - { - "cell_type": "code", - "execution_count": 47, - "id": "e0ab0af6-b5c9-4b74-9dd6-baa7737cc986", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "784" - ] - }, - "execution_count": 47, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "df = spark.read.parquet(\"fashion_mnist_784\")\n", - "len(df.columns)" - ] - }, - { - "cell_type": "code", - "execution_count": 48, - "id": "13ae45dc-85a0-4864-8a58-9dc29ae4efd7", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 1.08 s, sys: 142 ms, total: 1.23 s\n", - "Wall time: 6 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", mnist(struct(df.columns))).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 49, - "id": "0b3fb48b-f871-41f2-ac57-346899a6fe48", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 1.07 s, sys: 137 ms, total: 1.21 s\n", - "Wall time: 3.01 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", mnist(array(*df.columns))).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 50, - "id": "a43f3a7e-c6ef-4eaa-bfa8-8ca09cab7070", - "metadata": {}, - "outputs": [], - "source": [ - "# should raise ValueError\n", - "# preds = df.withColumn(\"preds\", mnist(*df.columns)).collect()" - ] - }, - { - "cell_type": "markdown", - "id": "dc48ec42-0df6-4e6a-b019-1270ab71d2cf", - "metadata": { - "tags": [] - }, - "source": [ - "### Check predictions" - ] - }, - { - "cell_type": "code", - "execution_count": 51, - "id": "d815c701-9f5b-422c-b3f9-fbc30456953c", - "metadata": {}, - "outputs": [], - "source": [ - "preds = df.withColumn(\"preds\", mnist(array(*df.columns))).limit(10).toPandas()" - ] - }, - { - "cell_type": "code", - "execution_count": 52, - "id": "b571b742-5079-42b2-8524-9181a0dec2c7", - "metadata": {}, - "outputs": [], - "source": [ - "sample = preds.iloc[0]\n", - "predictions = sample.preds\n", - "img = sample.drop('preds').to_numpy(dtype=float)" - ] - }, - { - "cell_type": "code", - "execution_count": 53, - "id": "d33d6a4e-e6b9-489d-ac21-c4eddc801784", - "metadata": {}, - "outputs": [], - "source": [ - "import matplotlib.pyplot as plt\n", - "import numpy as np" - ] - }, - { - "cell_type": "code", - "execution_count": 54, - "id": "6d10061e-aca6-4f81-bdfe-72e327ed7349", - "metadata": {}, - "outputs": [], - "source": [ - "img = np.array(img).reshape(28,28)" - ] - }, - { - "cell_type": "code", - "execution_count": 55, - "id": "01f70e08-2c1d-419f-8676-3f6f4aba760f", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAfwElEQVR4nO3df3DU9b3v8dfm10Iw2RBCfkmgARGsSGyppFSlWHKBOMcB5d7xV88FjwevNDhFtHrpqGjbmbR4xzp6qN6Z24KeEbXeERi9LR0EE8YasKAcSrUpSVOBkgTFZjck5ud+7h8c064Q4LNueCfh+ZjZGbL7feX75ptv8so3u/kk4JxzAgDgPEuyHgAAcGGigAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGAixXqAz4tGozp69KgyMjIUCASsxwEAeHLOqbW1VYWFhUpK6v86Z9AV0NGjR1VUVGQ9BgDgCzp8+LDGjRvX7+ODroAyMjIkSdfoeqUo1XgaAICvHnXrLf2q7+t5fwasgNatW6fHH39cTU1NKikp0dNPP62ZM2eeNffZj91SlKqUAAUEAEPOf64weranUQbkRQgvv/yyVq1apTVr1ujdd99VSUmJ5s+fr2PHjg3E7gAAQ9CAFNATTzyhZcuW6Y477tCXv/xlPfvss0pPT9cvfvGLgdgdAGAISngBdXV1ae/evSorK/v7TpKSVFZWppqamlO27+zsVCQSibkBAIa/hBfQxx9/rN7eXuXl5cXcn5eXp6amplO2r6ysVCgU6rvxCjgAuDCY/yLq6tWrFQ6H+26HDx+2HgkAcB4k/FVwOTk5Sk5OVnNzc8z9zc3Nys/PP2X7YDCoYDCY6DEAAINcwq+A0tLSNGPGDG3fvr3vvmg0qu3bt2vWrFmJ3h0AYIgakN8DWrVqlZYsWaKvfe1rmjlzpp588km1tbXpjjvuGIjdAQCGoAEpoJtvvlkfffSRHnnkETU1NenKK6/U1q1bT3lhAgDgwhVwzjnrIf5RJBJRKBTSHC1kJQQAGIJ6XLeqtEXhcFiZmZn9bmf+KjgAwIWJAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJhJeQI8++qgCgUDMberUqYneDQBgiEsZiHd6+eWX64033vj7TlIGZDcAgCFsQJohJSVF+fn5A/GuAQDDxIA8B3Tw4EEVFhZq4sSJuv3223Xo0KF+t+3s7FQkEom5AQCGv4QXUGlpqTZs2KCtW7fqmWeeUUNDg6699lq1traedvvKykqFQqG+W1FRUaJHAgAMQgHnnBvIHbS0tGjChAl64okndOedd57yeGdnpzo7O/vejkQiKioq0hwtVEogdSBHAwAMgB7XrSptUTgcVmZmZr/bDfirA7KysnTppZeqrq7utI8Hg0EFg8GBHgMAMMgM+O8BnThxQvX19SooKBjoXQEAhpCEF9D999+v6upq/eUvf9Hbb7+tG2+8UcnJybr11lsTvSsAwBCW8B/BHTlyRLfeequOHz+usWPH6pprrtGuXbs0duzYRO8KADCEJbyAXnrppUS/SwDAMMRacAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAEwM+B+kw38KBPwzcfyx2kCK/4fU9fR4Z07u7Pz8n4aj5NGjvTN/fHSKd2bStL96ZyQp/Pw478zo52ri2tdwkzRqlHcmns9B9w9/SXqo4goIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCC1bDPl/O0CnQ8q+rGs4J23JKTvSOuq8s7E4hjP9IXWBncU+OGPO9MoN5/P+V5f/APSZr28FbvzL/9y1zvTPP/KfbOZBzxXwU6+c13vTOSlDx5onemc7z/SueH56Z5Zyav+9A7I0k9fz3qH/Je+T4gncOXPK6AAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmGAx0uHGe9HA87cApyTpPO3rfP6fjj7wDe/MXZf8P+/ME03zvDNP1/gvECpJz1z3vHfmXwt3emd2rPzYO7P72ATvTPt0/4+RJI1qjnpn2vL8v6+/ruw978xbrV/xzkjSuEr/xUh9F/cNuKh0Dp+CXAEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwwWKkw41z1hOcUdf8r3ln0v/0kXemp+FD70y88n7X4Z1p/+c070zq3/w/XaN+a0j2ue8//qt35s4pNd6ZJwv2eGeK9/gvwjm2Jb7Pi5ZL/L9H75l2Iq59+fp0XO952c9A4goIAGCCAgIAmPAuoJ07d+qGG25QYWGhAoGANm/eHPO4c06PPPKICgoKNHLkSJWVlengwYOJmhcAMEx4F1BbW5tKSkq0bt260z6+du1aPfXUU3r22We1e/dujRo1SvPnz1dHh//PyQEAw5f3s5rl5eUqLy8/7WPOOT355JN66KGHtHDhQknS888/r7y8PG3evFm33HLLF5sWADBsJPQ5oIaGBjU1NamsrKzvvlAopNLSUtXUnP4VMp2dnYpEIjE3AMDwl9ACampqkiTl5eXF3J+Xl9f32OdVVlYqFAr13YqKihI5EgBgkDJ/Fdzq1asVDof7bocPH7YeCQBwHiS0gPLz8yVJzc3NMfc3Nzf3PfZ5wWBQmZmZMTcAwPCX0AIqLi5Wfn6+tm/f3ndfJBLR7t27NWvWrETuCgAwxHm/Cu7EiROqq6vre7uhoUH79u1Tdna2xo8fr5UrV+pHP/qRJk+erOLiYj388MMqLCzUokWLEjk3AGCI8y6gPXv26Lrrrut7e9WqVZKkJUuWaMOGDXrggQfU1tamu+66Sy0tLbrmmmu0detWjRgxInFTAwCGvIBzg2v1ykgkolAopDlaqJRA6sDuLBCILze4DtkXduT734grd1n5n7wzew9+yTtzU8m73pmt//fr3hlJGlf5dlw5X3/6uf+irFdfVnf2jT7nnarLvDOS1DvC/xy/9Hn/X6GovcP/Od/Rf/D/vI2mxvm5Pu8T70jbp/4LzfZ0+H+tm7x0r3cmbp5fK3tct6rcZoXD4TM+r2/+KjgAwIWJAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGDC+88xDFpxrGwdSE6Oa1eupyeunK+OG2Z6Zzb82xPemW3tx7wzklT59vXemaSw/ym36Q9Xemfu++fXvDOS9L/Gl3tnLl3+jn/mzj3emQ9eu9Q7kzrFf4VqSYo2ZHhnjqzx30/eK/6ZyAT/z/VR13zkvyNJHx3N8s4snflb78zuW6d5Z3q9E4MPV0AAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMBJxzznqIfxSJRBQKhTRHC5USSLUex9S3ft/mnfnlX77inQnXZntnJCm/xv/U6fqXT7wzbb8d650ZMetj74wkjUrr9s50P5fnncncuMs7E4+mzZedl/1IUuSTUd6ZoouPe2eOfpzlnUk6PMI7I0nBKWHvTPqrIe9M1r/XeGfiWYBZkhTPl3zPffW4blW5zQqHw8rMzOx3O66AAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmEixHsCSm1USVy6pJ+qd6bnIf2HV/13ln3Hpvd6ZlMIO74wkHb/Ff19zcw95Z7aNyvHOfNLkvyCkJEVzW70zXV/y/z6u/+UZEyt/0Qdx5T785RXemeyxEe/M4UP+H9v07HbvzLirPvLOSNLB9y/2zhTGsbBo8hkW7OxPtN3/OEiS6+mJI+S5gOk5bs8VEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABODdjHS6DXTFU0Zcc7btxYFvfcxqrHbOyNJ0ajnwnySjn3l3P8vn/mfZZu9Mz/+3QLvTHp6p3dGki4b2+yd2f9JoXemO9N/8df0P/sv5CpJSb8b7Z0Z2et/Ppz4b6Xema6L/L9fTDvhf+wkafxP/ReoPbzK/3PwkolN3pk/N/ovYDru4hbvjCQ11Y33zjTf8w3vTOoJ/3MoGInvYxuPzG1+i9o61yWdw9q0XAEBAExQQAAAE94FtHPnTt1www0qLCxUIBDQ5s2bYx5funSpAoFAzG3BAv8fCwEAhjfvAmpra1NJSYnWrVvX7zYLFixQY2Nj3+3FF1/8QkMCAIYf7xchlJeXq7y8/IzbBINB5efnxz0UAGD4G5DngKqqqpSbm6spU6Zo+fLlOn78eL/bdnZ2KhKJxNwAAMNfwgtowYIFev7557V9+3b95Cc/UXV1tcrLy9Xb23va7SsrKxUKhfpuRUVFiR4JADAIJfz3gG655Za+f19xxRWaPn26Jk2apKqqKs2dO/eU7VevXq1Vq1b1vR2JRCghALgADPjLsCdOnKicnBzV1dWd9vFgMKjMzMyYGwBg+BvwAjpy5IiOHz+ugoKCgd4VAGAI8f4R3IkTJ2KuZhoaGrRv3z5lZ2crOztbjz32mBYvXqz8/HzV19frgQce0CWXXKL58+cndHAAwNDmXUB79uzRdddd1/f2Z8/fLFmyRM8884z279+v5557Ti0tLSosLNS8efP0wx/+UMGg/zpRAIDhy7uA5syZI+f6XzjvN7/5zRca6DN/XpyqpJHnvqDkxW/4L+aXsuNd74wkdV7/Ne9MT7r/fhq7srwz11/2B+/Mx12jvDOS9B9HL/bOdB7131dyR8A70z6+xzsjSWPe98+0j032zoT+8DfvzInJWd6ZeB1a4P9xKsj6q3fmk/aR3pmLx7Z4Z/Y0xffCpouOnv7Vu2cSTfU/X7Pe9//1k8CnXd4ZSYpm+h/zhpXTvLbv7eyQfnz27VgLDgBgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgIuF/kjtRxu5OVnLaua8yPPl7B7z3cezd8d4ZSUr+03HvTOol+d6ZvS3+8/3+A/9M6if+qzlLUk8oGlfOV29Bp3cme/SJuPZ1/PIc78zFb7Z5Z6Kj/P88yUVvN3hn1Ol/7CTpyLe+7J1JS/JfOToeuemt3pmW38T3BzEzD7Z4ZwInPvXO9GZf5J1J9t+NJOnPi/33FU32+2sD0dRz254rIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYG7WKkoZfeUUog9Zy33zn/K977+Pkbz3lnJOm+n/wP70wgjnU7P2of5Z3JzPdfqDF9Qpd3RpJ6o/7fv3zS4r8QYm+7/2kajWM2SZrwX/7inakfNcE7M+b3fos7StLxRZd4Z8Z//Yh3RpKuTK33zvQ4/0Vts9PjXFHTU9b8xrhytVeO9s5Mm/CJd6Y5cu5f6z7T2pbunZGkqQX+i9p+8pTfOd7THdWH57AdV0AAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMBJxz/qsiDqBIJKJQKKQ5Wui1GGk8Om6YGVcu8q8R70xBpn/mvxfWeGfqOvO8Mx3R+I7ziKRu70wo2X/xyfqOsd6Zpo5M70y8jnf4Lxpb/8dC78yIRv/FPgtqOr0zkhQ89Df/UKr/orE9oZHemUCv/8q+nWNGeGckKZoW8M6kfOo/X1K3f6Yn3f98kKTudP/rjsxfHfDavsd1aUfbiwqHw8rM7P9zkSsgAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJvxXDxxGRrz2Tpw5/4ybNtU7s2HkP3lnOnL8F13siWNxQim+RQ1TOv3Xvk1t7fXOjGxs885IUtLfWv0zHx72zkyWf+Z88j/i8UlKieNLULL/IpzBri7//UhKzsiIK+cr2t7unUkbPTquffV+9JF3xnep1Kg7t4WKuQICAJiggAAAJrwKqLKyUldddZUyMjKUm5urRYsWqba2Nmabjo4OVVRUaMyYMbrooou0ePFiNTc3J3RoAMDQ51VA1dXVqqio0K5du7Rt2zZ1d3dr3rx5amv7+8/b7733Xr322mt65ZVXVF1draNHj+qmm25K+OAAgKHN6xnArVu3xry9YcMG5ebmau/evZo9e7bC4bB+/vOfa+PGjfrWt74lSVq/fr0uu+wy7dq1S1//+tcTNzkAYEj7Qs8BhcNhSVJ2drYkae/everu7lZZWVnfNlOnTtX48eNVU3P6Py/d2dmpSCQScwMADH9xF1A0GtXKlSt19dVXa9q0aZKkpqYmpaWlKSsrK2bbvLw8NTU1nfb9VFZWKhQK9d2KioriHQkAMITEXUAVFRU6cOCAXnrppS80wOrVqxUOh/tuhw8P7t+PAAAkRly/iLpixQq9/vrr2rlzp8aNG9d3f35+vrq6utTS0hJzFdTc3Kz8/PzTvq9gMKhgMBjPGACAIczrCsg5pxUrVmjTpk3asWOHiouLYx6fMWOGUlNTtX379r77amtrdejQIc2aNSsxEwMAhgWvK6CKigpt3LhRW7ZsUUZGRt/zOqFQSCNHjlQoFNKdd96pVatWKTs7W5mZmbrnnns0a9YsXgEHAIjhVUDPPPOMJGnOnDkx969fv15Lly6VJP30pz9VUlKSFi9erM7OTs2fP18/+9nPEjIsAGD4CDjn/FeHHECRSEShUEhztFApgVTrcQAAnnpct6q0ReFwWJmZmf1ux1pwAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBMUEAAABMUEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMOFVQJWVlbrqqquUkZGh3NxcLVq0SLW1tTHbzJkzR4FAIOZ29913J3RoAMDQ51VA1dXVqqio0K5du7Rt2zZ1d3dr3rx5amtri9lu2bJlamxs7LutXbs2oUMDAIa+FJ+Nt27dGvP2hg0blJubq71792r27Nl996enpys/Pz8xEwIAhqUv9BxQOByWJGVnZ8fc/8ILLygnJ0fTpk3T6tWr1d7e3u/76OzsVCQSibkBAIY/ryugfxSNRrVy5UpdffXVmjZtWt/9t912myZMmKDCwkLt379fDz74oGpra/Xqq6+e9v1UVlbqsccei3cMAMAQFXDOuXiCy5cv169//Wu99dZbGjduXL/b7dixQ3PnzlVdXZ0mTZp0yuOdnZ3q7OzsezsSiaioqEhztFApgdR4RgMAGOpx3arSFoXDYWVmZva7XVxXQCtWrNDrr7+unTt3nrF8JKm0tFSS+i2gYDCoYDAYzxgAgCHMq4Ccc7rnnnu0adMmVVVVqbi4+KyZffv2SZIKCgriGhAAMDx5FVBFRYU2btyoLVu2KCMjQ01NTZKkUCikkSNHqr6+Xhs3btT111+vMWPGaP/+/br33ns1e/ZsTZ8+fUD+AwCAocnrOaBAIHDa+9evX6+lS5fq8OHD+va3v60DBw6ora1NRUVFuvHGG/XQQw+d8eeA/ygSiSgUCvEcEAAMUQPyHNDZuqqoqEjV1dU+7xIAcIFiLTgAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgAkKCABgggICAJiggAAAJiggAIAJCggAYIICAgCYoIAAACYoIACACQoIAGCCAgIAmKCAAAAmKCAAgIkU6wE+zzknSepRt+SMhwEAeOtRt6S/fz3vz6AroNbWVknSW/qV8SQAgC+itbVVoVCo38cD7mwVdZ5Fo1EdPXpUGRkZCgQCMY9FIhEVFRXp8OHDyszMNJrQHsfhJI7DSRyHkzgOJw2G4+CcU2trqwoLC5WU1P8zPYPuCigpKUnjxo074zaZmZkX9An2GY7DSRyHkzgOJ3EcTrI+Dme68vkML0IAAJiggAAAJoZUAQWDQa1Zs0bBYNB6FFMch5M4DidxHE7iOJw0lI7DoHsRAgDgwjCkroAAAMMHBQQAMEEBAQBMUEAAABNDpoDWrVunL33pSxoxYoRKS0v1zjvvWI903j366KMKBAIxt6lTp1qPNeB27typG264QYWFhQoEAtq8eXPM4845PfLIIyooKNDIkSNVVlamgwcP2gw7gM52HJYuXXrK+bFgwQKbYQdIZWWlrrrqKmVkZCg3N1eLFi1SbW1tzDYdHR2qqKjQmDFjdNFFF2nx4sVqbm42mnhgnMtxmDNnzinnw91332008ekNiQJ6+eWXtWrVKq1Zs0bvvvuuSkpKNH/+fB07dsx6tPPu8ssvV2NjY9/trbfesh5pwLW1tamkpETr1q077eNr167VU089pWeffVa7d+/WqFGjNH/+fHV0dJznSQfW2Y6DJC1YsCDm/HjxxRfP44QDr7q6WhUVFdq1a5e2bdum7u5uzZs3T21tbX3b3HvvvXrttdf0yiuvqLq6WkePHtVNN91kOHXinctxkKRly5bFnA9r1641mrgfbgiYOXOmq6io6Hu7t7fXFRYWusrKSsOpzr81a9a4kpIS6zFMSXKbNm3qezsajbr8/Hz3+OOP993X0tLigsGge/HFFw0mPD8+fxycc27JkiVu4cKFJvNYOXbsmJPkqqurnXMnP/apqanulVde6dvmgw8+cJJcTU2N1ZgD7vPHwTnnvvnNb7rvfve7dkOdg0F/BdTV1aW9e/eqrKys776kpCSVlZWppqbGcDIbBw8eVGFhoSZOnKjbb79dhw4dsh7JVENDg5qammLOj1AopNLS0gvy/KiqqlJubq6mTJmi5cuX6/jx49YjDahwOCxJys7OliTt3btX3d3dMefD1KlTNX78+GF9Pnz+OHzmhRdeUE5OjqZNm6bVq1ervb3dYrx+DbrFSD/v448/Vm9vr/Ly8mLuz8vL0x//+EejqWyUlpZqw4YNmjJlihobG/XYY4/p2muv1YEDB5SRkWE9nommpiZJOu358dljF4oFCxbopptuUnFxserr6/X9739f5eXlqqmpUXJysvV4CReNRrVy5UpdffXVmjZtmqST50NaWpqysrJith3O58PpjoMk3XbbbZowYYIKCwu1f/9+Pfjgg6qtrdWrr75qOG2sQV9A+Lvy8vK+f0+fPl2lpaWaMGGCfvnLX+rOO+80nAyDwS233NL37yuuuELTp0/XpEmTVFVVpblz5xpONjAqKip04MCBC+J50DPp7zjcddddff++4oorVFBQoLlz56q+vl6TJk0632Oe1qD/EVxOTo6Sk5NPeRVLc3Oz8vPzjaYaHLKysnTppZeqrq7OehQzn50DnB+nmjhxonJycobl+bFixQq9/vrrevPNN2P+fEt+fr66urrU0tISs/1wPR/6Ow6nU1paKkmD6nwY9AWUlpamGTNmaPv27X33RaNRbd++XbNmzTKczN6JEydUX1+vgoIC61HMFBcXKz8/P+b8iEQi2r179wV/fhw5ckTHjx8fVueHc04rVqzQpk2btGPHDhUXF8c8PmPGDKWmpsacD7W1tTp06NCwOh/OdhxOZ9++fZI0uM4H61dBnIuXXnrJBYNBt2HDBvf++++7u+66y2VlZbmmpibr0c6r++67z1VVVbmGhgb329/+1pWVlbmcnBx37Ngx69EGVGtrq3vvvffce++95yS5J554wr333nvuww8/dM459+Mf/9hlZWW5LVu2uP3797uFCxe64uJi9+mnnxpPnlhnOg6tra3u/vvvdzU1Na6hocG98cYb7qtf/aqbPHmy6+josB49YZYvX+5CoZCrqqpyjY2Nfbf29va+be6++243fvx4t2PHDrdnzx43a9YsN2vWLMOpE+9sx6Gurs794Ac/cHv27HENDQ1uy5YtbuLEiW727NnGk8caEgXknHNPP/20Gz9+vEtLS3MzZ850u3btsh7pvLv55ptdQUGBS0tLcxdffLG7+eabXV1dnfVYA+7NN990kk65LVmyxDl38qXYDz/8sMvLy3PBYNDNnTvX1dbW2g49AM50HNrb2928efPc2LFjXWpqqpswYYJbtmzZsPsm7XT/f0lu/fr1fdt8+umn7jvf+Y4bPXq0S09PdzfeeKNrbGy0G3oAnO04HDp0yM2ePdtlZ2e7YDDoLrnkEve9733PhcNh28E/hz/HAAAwMeifAwIADE8UEADABAUEADBBAQEATFBAAAATFBAAwAQFBAAwQQEBAExQQAAAExQQAMAEBQQAMEEBAQBM/H+wVOxLQwVbkwAAAABJRU5ErkJggg==", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "plt.figure()\n", - "plt.imshow(img)\n", - "plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 56, - "id": "8e1c07cc-b2bc-4902-a9a6-4ac7f02c5fe4", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[-1.720082402229309,\n", - " -2.013301134109497,\n", - " -0.3198302984237671,\n", - " -1.4132847785949707,\n", - " -0.25933507084846497,\n", - " 1.7492954730987549,\n", - " -0.4581696689128876,\n", - " 2.1313018798828125,\n", - " 1.6222645044326782,\n", - " 1.3297783136367798]" - ] - }, - "execution_count": 56, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "predictions" - ] - }, - { - "cell_type": "markdown", - "id": "a937adc9-508d-4ccd-b92d-8ecaa27ee4e4", - "metadata": {}, - "source": [ - "### Using Triton Inference Server\n", - "\n", - "Note: you can restart the kernel and run from this point to simulate running in a different node or environment." - ] - }, - { - "cell_type": "code", - "execution_count": 57, - "id": "53ca290a-ccc3-4923-a292-944921bab36d", - "metadata": {}, - "outputs": [], - "source": [ - "import numpy as np\n", - "\n", - "from functools import partial\n", - "from pyspark.ml.functions import predict_batch_udf\n", - "from pyspark.sql.functions import struct, col, array\n", - "from pyspark.sql.types import ArrayType, FloatType, Union, Dict" - ] - }, - { - "cell_type": "code", - "execution_count": 58, - "id": "8fa92fe4-2e04-4d82-a357-bfdfca38bd8c", - "metadata": {}, - "outputs": [], - "source": [ - "%%bash\n", - "# copy model to expected layout for Triton\n", - "rm -rf models\n", - "mkdir -p models/fashion_mnist/1\n", - "cp model.ts models/fashion_mnist/1/model.pt\n", - "\n", - "# add config.pbtxt\n", - "cp models_config/fashion_mnist/config.pbtxt models/fashion_mnist/config.pbtxt" - ] - }, - { - "cell_type": "markdown", - "id": "d42b329c-5921-436f-bfca-a382a6762da4", - "metadata": {}, - "source": [ - "#### Start Triton Server on each executor" - ] - }, - { - "cell_type": "code", - "execution_count": 59, - "id": "5e869730-3597-4074-bab0-f87768f8996a", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "data": { - "text/plain": [ - "[True]" - ] - }, - "execution_count": 59, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "num_executors = 1\n", - "triton_models_dir = \"{}/models\".format(os.getcwd())\n", - "nodeRDD = sc.parallelize(list(range(num_executors)), num_executors)\n", - "\n", - "def start_triton(it):\n", - " import docker\n", - " import time\n", - " import tritonclient.grpc as grpcclient\n", - " \n", - " client=docker.from_env()\n", - " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", - " if containers:\n", - " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", - " else:\n", - " container=client.containers.run(\n", - " \"nvcr.io/nvidia/tritonserver:22.07-py3\", \"tritonserver --model-repository=/models\",\n", - " detach=True,\n", - " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", - " name=\"spark-triton\",\n", - " network_mode=\"host\",\n", - " remove=True,\n", - " shm_size=\"64M\",\n", - " volumes={triton_models_dir: {\"bind\": \"/models\", \"mode\": \"ro\"}}\n", - " )\n", - " print(\">>>> starting triton: {}\".format(container.short_id))\n", - "\n", - " # wait for triton to be running\n", - " time.sleep(15)\n", - " client = grpcclient.InferenceServerClient(\"localhost:8001\")\n", - " ready = False\n", - " while not ready:\n", - " try:\n", - " ready = client.is_server_ready()\n", - " except Exception as e:\n", - " time.sleep(5)\n", - " \n", - " return [True]\n", - "\n", - "nodeRDD.barrier().mapPartitions(start_triton).collect()" - ] - }, - { - "cell_type": "markdown", - "id": "30a4362d-7514-4b84-b238-f704a97e1e72", - "metadata": {}, - "source": [ - "#### Run inference" - ] - }, - { - "cell_type": "code", - "execution_count": 60, - "id": "ab94d4d1-dac6-4474-9eb0-59478aa98f7d", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "1" - ] - }, - "execution_count": 60, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "df = spark.read.parquet(\"fashion_mnist_1\")\n", - "len(df.columns)" - ] - }, - { - "cell_type": "code", - "execution_count": 61, - "id": "12b5f2fc-52e9-428a-b683-6ab1b639aa24", - "metadata": { - "scrolled": true - }, - "outputs": [ - { - "data": { - "text/plain": [ - "StructType([StructField('data', ArrayType(FloatType(), True), True)])" - ] - }, - "execution_count": 61, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "df.schema" - ] - }, - { - "cell_type": "code", - "execution_count": 62, - "id": "960657d0-31c9-4df6-8eb8-ac3d23137f7a", - "metadata": {}, - "outputs": [], - "source": [ - "def triton_fn(triton_uri, model_name):\n", - " import numpy as np\n", - " import tritonclient.grpc as grpcclient\n", - " \n", - " np_types = {\n", - " \"BOOL\": np.dtype(np.bool8),\n", - " \"INT8\": np.dtype(np.int8),\n", - " \"INT16\": np.dtype(np.int16),\n", - " \"INT32\": np.dtype(np.int32),\n", - " \"INT64\": np.dtype(np.int64),\n", - " \"FP16\": np.dtype(np.float16),\n", - " \"FP32\": np.dtype(np.float32),\n", - " \"FP64\": np.dtype(np.float64),\n", - " \"FP64\": np.dtype(np.double),\n", - " \"BYTES\": np.dtype(object)\n", - " }\n", - "\n", - " client = grpcclient.InferenceServerClient(triton_uri)\n", - " model_meta = client.get_model_metadata(model_name)\n", - " \n", - " def predict(inputs):\n", - " if isinstance(inputs, np.ndarray):\n", - " # single ndarray input\n", - " request = [grpcclient.InferInput(model_meta.inputs[0].name, inputs.shape, model_meta.inputs[0].datatype)]\n", - " request[0].set_data_from_numpy(inputs.astype(np_types[model_meta.inputs[0].datatype]))\n", - " else:\n", - " # dict of multiple ndarray inputs\n", - " request = [grpcclient.InferInput(i.name, inputs[i.name].shape, i.datatype) for i in model_meta.inputs]\n", - " for i in request:\n", - " i.set_data_from_numpy(inputs[i.name()].astype(np_types[i.datatype()]))\n", - " \n", - " response = client.infer(model_name, inputs=request)\n", - " \n", - " if len(model_meta.outputs) > 1:\n", - " # return dictionary of numpy arrays\n", - " return {o.name: response.as_numpy(o.name) for o in model_meta.outputs}\n", - " else:\n", - " # return single numpy array\n", - " return response.as_numpy(model_meta.outputs[0].name)\n", - " \n", - " return predict" - ] - }, - { - "cell_type": "code", - "execution_count": 63, - "id": "0262fd4a-9845-44b9-8c75-1c105e7deeca", - "metadata": {}, - "outputs": [], - "source": [ - "mnist = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"fashion_mnist\"),\n", - " input_tensor_shapes=[[784]],\n", - " return_type=ArrayType(FloatType()),\n", - " batch_size=1024)" - ] - }, - { - "cell_type": "code", - "execution_count": 64, - "id": "fc5f6baa-052e-4b89-94b6-4821cf01952a", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 491 ms, sys: 60.7 ms, total: 551 ms\n", - "Wall time: 1.98 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", mnist(struct(df.columns))).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 65, - "id": "a85dea35-e41d-482d-8a8f-52d3c108f038", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 817 ms, sys: 89 ms, total: 906 ms\n", - "Wall time: 1.44 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", mnist(*df.columns)).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 66, - "id": "bc3f0dbe-c52b-41d6-8097-8cebaa5ee5a8", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 460 ms, sys: 105 ms, total: 565 ms\n", - "Wall time: 1.11 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", mnist(*[col(c) for c in df.columns])).collect()" - ] - }, - { - "cell_type": "markdown", - "id": "7a26690a-9dc4-4c36-9904-568d73e2be3c", - "metadata": { - "tags": [] - }, - "source": [ - "#### Stop Triton Server on each executor" - ] - }, - { - "cell_type": "code", - "execution_count": 67, - "id": "ab2fe42f-a072-4370-bac2-52fd95363530", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "data": { - "text/plain": [ - "[True]" - ] - }, - "execution_count": 67, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "def stop_triton(it):\n", - " import docker\n", - " import time\n", - " \n", - " client=docker.from_env()\n", - " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", - " print(\">>>> stopping containers: {}\".format([c.short_id for c in containers]))\n", - " if containers:\n", - " container=containers[0]\n", - " container.stop(timeout=120)\n", - "\n", - " return [True]\n", - "\n", - "nodeRDD.barrier().mapPartitions(stop_triton).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 68, - "id": "a0608fff-7cfb-489e-96c9-8e1d92e57562", - "metadata": {}, - "outputs": [], - "source": [ - "spark.stop()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "08de2664-3d60-487b-90da-6d0f3b8b9203", - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/image_classification_torch.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/image_classification_torch.ipynb new file mode 100644 index 000000000..47489dccb --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/image_classification_torch.ipynb @@ -0,0 +1,2367 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "9e87c927", + "metadata": {}, + "source": [ + "# PySpark PyTorch Inference\n", + "\n", + "### Image Classification\n", + "Based on: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html \n", + "\n", + "Also demonstrates accelerated inference on GPU with Torch-TensorRT. " + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "91d7ec98", + "metadata": {}, + "outputs": [], + "source": [ + "import torch\n", + "\n", + "from torch import nn\n", + "from torch.utils.data import DataLoader\n", + "from torchvision import datasets\n", + "from torchvision.transforms import ToTensor" + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "d714f40d", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "'2.4.1+cu121'" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "torch.__version__" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "1c942a46", + "metadata": {}, + "outputs": [], + "source": [ + "# Download training data from open datasets.\n", + "training_data = datasets.FashionMNIST(\n", + " root=\"data\",\n", + " train=True,\n", + " download=True,\n", + " transform=ToTensor(),\n", + ")\n", + "\n", + "# Download test data from open datasets.\n", + "test_data = datasets.FashionMNIST(\n", + " root=\"data\",\n", + " train=False,\n", + " download=True,\n", + " transform=ToTensor(),\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "4a89aa8e-ef62-4aac-8260-4b004f2c1b55", + "metadata": {}, + "outputs": [], + "source": [ + "classes = [\n", + " \"T-shirt/top\",\n", + " \"Trouser\",\n", + " \"Pullover\",\n", + " \"Dress\",\n", + " \"Coat\",\n", + " \"Sandal\",\n", + " \"Shirt\",\n", + " \"Sneaker\",\n", + " \"Bag\",\n", + " \"Ankle boot\",\n", + "]" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "10a97111", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Shape of X [N, C, H, W]: torch.Size([64, 1, 28, 28]) torch.float32\n", + "Shape of y: torch.Size([64]) torch.int64\n" + ] + } + ], + "source": [ + "batch_size = 64\n", + "\n", + "# Create data loaders.\n", + "train_dataloader = DataLoader(training_data, batch_size=batch_size)\n", + "test_dataloader = DataLoader(test_data, batch_size=batch_size)\n", + "\n", + "for X, y in test_dataloader:\n", + " print(f\"Shape of X [N, C, H, W]: {X.shape} {X.dtype}\")\n", + " print(f\"Shape of y: {y.shape} {y.dtype}\")\n", + " break" + ] + }, + { + "cell_type": "markdown", + "id": "ca7af350", + "metadata": {}, + "source": [ + "### Create model" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "512d0bc7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Using cuda device\n", + "NeuralNetwork(\n", + " (linear_relu_stack): Sequential(\n", + " (0): Linear(in_features=784, out_features=512, bias=True)\n", + " (1): ReLU()\n", + " (2): Linear(in_features=512, out_features=512, bias=True)\n", + " (3): ReLU()\n", + " (4): Linear(in_features=512, out_features=10, bias=True)\n", + " )\n", + ")\n" + ] + } + ], + "source": [ + "# Get cpu or gpu device for training.\n", + "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + "print(f\"Using {device} device\")\n", + "\n", + "# Define model\n", + "class NeuralNetwork(nn.Module):\n", + " def __init__(self):\n", + " super(NeuralNetwork, self).__init__()\n", + " self.linear_relu_stack = nn.Sequential(\n", + " nn.Linear(28*28, 512),\n", + " nn.ReLU(),\n", + " nn.Linear(512, 512),\n", + " nn.ReLU(),\n", + " nn.Linear(512, 10)\n", + " )\n", + "\n", + " def forward(self, x):\n", + " logits = self.linear_relu_stack(x)\n", + " return logits\n", + "\n", + "model = NeuralNetwork().to(device)\n", + "print(model)" + ] + }, + { + "cell_type": "markdown", + "id": "4573c1b7", + "metadata": {}, + "source": [ + "### Train Model" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "4d4f5538", + "metadata": {}, + "outputs": [], + "source": [ + "loss_fn = nn.CrossEntropyLoss()\n", + "optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "92d9076a", + "metadata": {}, + "outputs": [], + "source": [ + "def train(dataloader, model, loss_fn, optimizer):\n", + " size = len(dataloader.dataset)\n", + " model.train()\n", + " for batch, (X, y) in enumerate(dataloader):\n", + " X, y = X.to(device), y.to(device)\n", + " X = torch.flatten(X, start_dim=1, end_dim=-1)\n", + "\n", + " # Zero gradients\n", + " optimizer.zero_grad()\n", + "\n", + " # Compute prediction error\n", + " pred = model(X)\n", + " loss = loss_fn(pred, y)\n", + "\n", + " # Backpropagation\n", + " loss.backward()\n", + " optimizer.step()\n", + "\n", + " if batch % 100 == 0:\n", + " loss, current = loss.item(), (batch + 1) * len(X)\n", + " print(f\"loss: {loss:>7f} [{current:>5d}/{size:>5d}]\")" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "11c5650d", + "metadata": {}, + "outputs": [], + "source": [ + "def test(dataloader, model, loss_fn):\n", + " size = len(dataloader.dataset)\n", + " num_batches = len(dataloader)\n", + " model.eval()\n", + " test_loss, correct = 0, 0\n", + " with torch.no_grad():\n", + " for X, y in dataloader:\n", + " X, y = X.to(device), y.to(device)\n", + " X = torch.flatten(X, start_dim=1, end_dim=-1)\n", + " pred = model(X)\n", + " test_loss += loss_fn(pred, y).item()\n", + " correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n", + " test_loss /= num_batches\n", + " correct /= size\n", + " print(f\"Test Error: \\n Accuracy: {(100*correct):>0.1f}%, Avg loss: {test_loss:>8f} \\n\")" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "854608e6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1\n", + "-------------------------------\n", + "loss: 2.299719 [ 64/60000]\n", + "loss: 2.293332 [ 6464/60000]\n", + "loss: 2.269917 [12864/60000]\n", + "loss: 2.260744 [19264/60000]\n", + "loss: 2.247810 [25664/60000]\n", + "loss: 2.222256 [32064/60000]\n", + "loss: 2.225422 [38464/60000]\n", + "loss: 2.195026 [44864/60000]\n", + "loss: 2.194622 [51264/60000]\n", + "loss: 2.158175 [57664/60000]\n", + "Test Error: \n", + " Accuracy: 47.1%, Avg loss: 2.153042 \n", + "\n", + "Epoch 2\n", + "-------------------------------\n", + "loss: 2.162534 [ 64/60000]\n", + "loss: 2.154336 [ 6464/60000]\n", + "loss: 2.091042 [12864/60000]\n", + "loss: 2.104471 [19264/60000]\n", + "loss: 2.054451 [25664/60000]\n", + "loss: 2.001035 [32064/60000]\n", + "loss: 2.025180 [38464/60000]\n", + "loss: 1.949615 [44864/60000]\n", + "loss: 1.957106 [51264/60000]\n", + "loss: 1.876436 [57664/60000]\n", + "Test Error: \n", + " Accuracy: 54.6%, Avg loss: 1.876885 \n", + "\n", + "Epoch 3\n", + "-------------------------------\n", + "loss: 1.906243 [ 64/60000]\n", + "loss: 1.879715 [ 6464/60000]\n", + "loss: 1.758657 [12864/60000]\n", + "loss: 1.795318 [19264/60000]\n", + "loss: 1.692177 [25664/60000]\n", + "loss: 1.652430 [32064/60000]\n", + "loss: 1.669603 [38464/60000]\n", + "loss: 1.583420 [44864/60000]\n", + "loss: 1.603508 [51264/60000]\n", + "loss: 1.493881 [57664/60000]\n", + "Test Error: \n", + " Accuracy: 61.9%, Avg loss: 1.514976 \n", + "\n", + "Epoch 4\n", + "-------------------------------\n", + "loss: 1.573342 [ 64/60000]\n", + "loss: 1.548722 [ 6464/60000]\n", + "loss: 1.402007 [12864/60000]\n", + "loss: 1.461628 [19264/60000]\n", + "loss: 1.353920 [25664/60000]\n", + "loss: 1.358175 [32064/60000]\n", + "loss: 1.361608 [38464/60000]\n", + "loss: 1.302804 [44864/60000]\n", + "loss: 1.330850 [51264/60000]\n", + "loss: 1.224925 [57664/60000]\n", + "Test Error: \n", + " Accuracy: 63.8%, Avg loss: 1.254037 \n", + "\n", + "Epoch 5\n", + "-------------------------------\n", + "loss: 1.321162 [ 64/60000]\n", + "loss: 1.315946 [ 6464/60000]\n", + "loss: 1.152864 [12864/60000]\n", + "loss: 1.244943 [19264/60000]\n", + "loss: 1.130193 [25664/60000]\n", + "loss: 1.160290 [32064/60000]\n", + "loss: 1.168214 [38464/60000]\n", + "loss: 1.123758 [44864/60000]\n", + "loss: 1.158085 [51264/60000]\n", + "loss: 1.063427 [57664/60000]\n", + "Test Error: \n", + " Accuracy: 65.1%, Avg loss: 1.089330 \n", + "\n", + "Done!\n" + ] + } + ], + "source": [ + "epochs = 5\n", + "for t in range(epochs):\n", + " print(f\"Epoch {t+1}\\n-------------------------------\")\n", + " train(train_dataloader, model, loss_fn, optimizer)\n", + " test(test_dataloader, model, loss_fn)\n", + "print(\"Done!\")" + ] + }, + { + "cell_type": "markdown", + "id": "85d97839", + "metadata": {}, + "source": [ + "### Save Model State Dict\n", + "This saves the serialized object to disk using pickle." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "5d5d24de", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saved PyTorch Model State to model.pt\n" + ] + } + ], + "source": [ + "torch.save(model.state_dict(), \"model.pt\")\n", + "print(\"Saved PyTorch Model State to model.pt\")" + ] + }, + { + "cell_type": "markdown", + "id": "ac221ca7-e227-4c8c-8577-1eeda4a61fc7", + "metadata": {}, + "source": [ + "### Save Model as TorchScript\n", + "This saves an [intermediate representation of the compute graph](https://pytorch.org/tutorials/beginner/saving_loading_models.html#export-load-model-in-torchscript-format), which does not require pickle (or even python). " + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "6d9b3a45-7618-43e4-8bd3-8bb317a484d3", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saved TorchScript Model to ts_model.pt\n" + ] + } + ], + "source": [ + "scripted = torch.jit.script(model)\n", + "scripted.save(\"ts_model.pt\")\n", + "print(\"Saved TorchScript Model to ts_model.pt\")" + ] + }, + { + "cell_type": "markdown", + "id": "12ee8916-f437-4a2a-9bf4-14ff5376d305", + "metadata": {}, + "source": [ + "### Load Model State" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "8fe3b5d1", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "model_from_state = NeuralNetwork().to(device)\n", + "model_from_state.load_state_dict(torch.load(\"model.pt\", weights_only=True))" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "0c405bd0", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Predicted: \"Ankle boot\", Actual: \"Ankle boot\"\n" + ] + } + ], + "source": [ + "model_from_state.eval()\n", + "x, y = test_data[0][0], test_data[0][1]\n", + "with torch.no_grad():\n", + " x = torch.flatten(x.to(device), start_dim=1, end_dim=-1)\n", + " pred = model_from_state(x)\n", + " predicted, actual = classes[pred[0].argmax(0)], classes[y]\n", + " print(f'Predicted: \"{predicted}\", Actual: \"{actual}\"')" + ] + }, + { + "cell_type": "markdown", + "id": "290c482a-1c5d-4bf2-bc3f-8a4e53d442b5", + "metadata": {}, + "source": [ + "### Load Torchscript Model" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "ef3c419e-d384-446c-b07b-1af93e07d6c0", + "metadata": {}, + "outputs": [], + "source": [ + "# Load model to original device (GPU) and move to CPU. \n", + "ts_model = torch.jit.load(\"ts_model.pt\")" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "c92d6cdb", + "metadata": {}, + "outputs": [], + "source": [ + "x, y = test_data[0][0], test_data[0][1]" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "038af830-a360-45eb-ab4e-b1adab0af164", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Predicted: \"Ankle boot\", Actual: \"Ankle boot\"\n" + ] + } + ], + "source": [ + "with torch.no_grad():\n", + " pred = ts_model(torch.flatten(x.to(device), start_dim=1, end_dim=-1))\n", + " predicted, actual = classes[pred[0].argmax(0)], classes[y]\n", + " print(f'Predicted: \"{predicted}\", Actual: \"{actual}\"')" + ] + }, + { + "cell_type": "markdown", + "id": "76980495", + "metadata": {}, + "source": [ + "### Compile using the Torch JIT Compiler\n", + "This leverages the [Torch-TensorRT inference compiler](https://pytorch.org/TensorRT/) for accelerated inference on GPUs using the `torch.compile` JIT interface under the hood. The compiler stack returns a [boxed-function](http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/) that triggers compilation on the first call. \n", + "\n", + "Modules compiled in this fashion are [not serializable with pickle](https://github.com/pytorch/pytorch/issues/101107#issuecomment-1542688089), so we cannot send the compiled model directly to Spark. Instead, we will recompile and cache the model on the executor. " + ] + }, + { + "cell_type": "markdown", + "id": "414bc856", + "metadata": {}, + "source": [ + "(You may see a warning about modelopt quantization. This is safe to ignore, as [implicit quantization](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#intro-quantization) is deprecated in the latest TensorRT. See [this link](https://pytorch.org/TensorRT/tutorials/_rendered_examples/dynamo/vgg16_fp8_ptq.html) for a guide to explicit quantization.)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "f3e3bdc4", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING:torch_tensorrt.dynamo.conversion.aten_ops_converters:Unable to import quantization op. Please install modelopt library (https://github.com/NVIDIA/TensorRT-Model-Optimizer?tab=readme-ov-file#installation) to add support for compiling quantized models\n", + "INFO:torch_tensorrt.dynamo.utils:Using Default Torch-TRT Runtime (as requested by user)\n", + "INFO:torch_tensorrt.dynamo.utils:Device not specified, using Torch default current device - cuda:0. If this is incorrect, please specify an input device, via the device keyword.\n", + "INFO:torch_tensorrt.dynamo.utils:Compilation Settings: CompilationSettings(enabled_precisions={}, debug=False, workspace_size=0, min_block_size=5, torch_executed_ops=set(), pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, assume_dynamic_shape_support=False, sparse_weights=False, refit=False, engine_capability=, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912, dryrun=False, hardware_compatible=False, timing_cache_path='/tmp/timing_cache.bin')\n", + "\n", + "WARNING:torch_tensorrt.dynamo._compiler:Node _param_constant1 of op type get_attr does not have metadata. This could sometimes lead to undefined behavior.\n", + "WARNING:torch_tensorrt.dynamo._compiler:Some nodes do not have metadata (shape and dtype information). This could lead to problems sometimes if the graph has PyTorch and TensorRT segments.\n", + "INFO:torch_tensorrt.dynamo._compiler:Partitioning the graph via the fast partitioner\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 457, GPU 713 (MiB)\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init builder kernel library: CPU +1634, GPU +288, now: CPU 2238, GPU 1001 (MiB)\n", + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/torch_tensorrt/dynamo/conversion/impl/activation/base.py:40: DeprecationWarning: Use Deprecated in TensorRT 10.1. Superseded by explicit quantization. instead.\n", + " if input_val.dynamic_range is not None and dyn_range_fn is not None:\n", + "\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT INetwork construction elapsed time: 0:00:00.005708\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Global timing cache in use. Profiling results in this builder pass will be stored.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Detected 1 inputs and 1 output network tensors.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Host Persistent Memory: 21984\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Device Persistent Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Scratch Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Started assigning block shifts. This will take 4 steps to complete.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Algorithm ShiftNTopDown took 0.257559ms to assign 2 blocks to 4 nodes requiring 4096 bytes.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Activation Memory: 4096\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Weights Memory: 2678824\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Engine generation completed in 0.023755 seconds.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 1 MiB, GPU 5 MiB\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage during Engine building and serialization: CPU: 3800 MiB\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:Build TRT engine elapsed time: 0:00:00.027501\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT Engine uses: 2832188 bytes of Memory\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 26 bytes of code generator cache.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 176 timing cache entries\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Predicted: \"Ankle boot\", Actual: \"Ankle boot\"\n" + ] + } + ], + "source": [ + "import torch_tensorrt as trt\n", + "\n", + "inputs_bs1 = torch.randn((1, 784), dtype=torch.float).to(\"cuda\")\n", + "# This indicates dimension 0 of inputs_bs1 is dynamic whose range of values is [1, 50]. No recompilation will happen when the batch size changes.\n", + "torch._dynamo.mark_dynamic(inputs_bs1, 0, min=1, max=64)\n", + "trt_model = trt.compile(\n", + " model,\n", + " ir=\"torch_compile\",\n", + " inputs=inputs_bs1,\n", + " enabled_precisions={torch.float},\n", + ")\n", + "\n", + "stream = torch.cuda.Stream()\n", + "with torch.no_grad(), torch.cuda.stream(stream):\n", + " pred = trt_model(torch.flatten(x.to(device), start_dim=1, end_dim=-1))\n", + " predicted, actual = classes[pred[0].argmax(0)], classes[y]\n", + " print(f'Predicted: \"{predicted}\", Actual: \"{actual}\"')" + ] + }, + { + "cell_type": "markdown", + "id": "9ec04be8", + "metadata": {}, + "source": [ + "### Compile using the Torch-TensorRT AOT Compiler\n", + "Alternatively, use the Torch-TensorRT Dynamo backend for Ahead-of-Time (AOT) compilation to eagerly optimize the model in an explicit compilation phase. We first export the model to produce a traced graph representing the Tensor computation in an AOT fashion, which produces a `ExportedProgram` object which can be [serialized and reloaded](https://pytorch.org/TensorRT/user_guide/saving_models.html). We can then compile this IR using the Torch-TensorRT AOT compiler for inference. \n", + "\n", + "[Read the docs](https://pytorch.org/TensorRT/user_guide/torch_tensorrt_explained.html) for more information on JIT vs AOT compilation." + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "6b8f1b45", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:torch_tensorrt.dynamo._compiler:Compilation Settings: CompilationSettings(enabled_precisions={}, debug=False, workspace_size=0, min_block_size=5, torch_executed_ops=set(), pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=True, disable_tf32=False, assume_dynamic_shape_support=False, sparse_weights=False, refit=False, engine_capability=, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912, dryrun=False, hardware_compatible=False, timing_cache_path='/tmp/timing_cache.bin')\n", + "\n", + "INFO:torch_tensorrt.dynamo._compiler:Partitioning the graph via the fast partitioner\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 624, GPU 715 (MiB)\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init builder kernel library: CPU +1632, GPU +286, now: CPU 2256, GPU 1001 (MiB)\n", + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/torch_tensorrt/dynamo/conversion/impl/activation/base.py:40: DeprecationWarning: Use Deprecated in TensorRT 10.1. Superseded by explicit quantization. instead.\n", + " if input_val.dynamic_range is not None and dyn_range_fn is not None:\n", + "\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT INetwork construction elapsed time: 0:00:00.004551\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Global timing cache in use. Profiling results in this builder pass will be stored.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Detected 1 inputs and 1 output network tensors.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Host Persistent Memory: 21984\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Device Persistent Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Scratch Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Started assigning block shifts. This will take 4 steps to complete.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Algorithm ShiftNTopDown took 0.133258ms to assign 2 blocks to 4 nodes requiring 4096 bytes.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Activation Memory: 4096\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Weights Memory: 2678824\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Engine generation completed in 0.0190609 seconds.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 1 MiB, GPU 5 MiB\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage during Engine building and serialization: CPU: 3818 MiB\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:Build TRT engine elapsed time: 0:00:00.021306\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT Engine uses: 2833124 bytes of Memory\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 26 bytes of code generator cache.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 176 timing cache entries\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Predicted: \"Ankle boot\", Actual: \"Ankle boot\"\n" + ] + } + ], + "source": [ + "# Preparing the inputs for batch_size = 1. \n", + "inputs = (torch.randn((1, 784), dtype=torch.float).cuda(),)\n", + "\n", + "# Produce traced graph in the ExportedProgram format\n", + "exp_program = trt.dynamo.trace(model_from_state, inputs)\n", + "# Compile the traced graph to produce an optimized module\n", + "trt_gm = trt.dynamo.compile(exp_program, inputs=inputs, require_full_compilation=True)\n", + "\n", + "stream = torch.cuda.Stream()\n", + "with torch.no_grad(), torch.cuda.stream(stream):\n", + " trt_gm(torch.flatten(x.to(device), start_dim=1, end_dim=-1))\n", + " predicted, actual = classes[pred[0].argmax(0)], classes[y]\n", + " print(f'Predicted: \"{predicted}\", Actual: \"{actual}\"')" + ] + }, + { + "cell_type": "markdown", + "id": "6f2bbfe1", + "metadata": {}, + "source": [ + "We can save the compiled model using `torch_tensorrt.save`. Unfortunately, serializing the model to be reloaded at a later date currently only supports *static inputs*." + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "d87e4b20", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/torch_tensorrt/dynamo/_exporter.py:364: UserWarning: Attempted to insert a get_attr Node with no underlying reference in the owning GraphModule! Call GraphModule.add_submodule to add the necessary submodule, GraphModule.add_parameter to add the necessary Parameter, or nn.Module.register_buffer to add the necessary buffer\n", + " engine_node = gm.graph.get_attr(engine_name)\n", + "\n", + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/torch/fx/graph.py:1545: UserWarning: Node _run_on_acc_0_engine target _run_on_acc_0_engine _run_on_acc_0_engine of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target\n", + " warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '\n", + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saved AOT compiled TensorRT model to trt_model_aot.ep\n" + ] + } + ], + "source": [ + "with torch.cuda.stream(stream):\n", + " trt.save(trt_gm, \"trt_model_aot.ep\", inputs=[torch.randn((1, 784), dtype=torch.float).to(\"cuda\")])\n", + " print(\"Saved AOT compiled TensorRT model to trt_model_aot.ep\")" + ] + }, + { + "cell_type": "markdown", + "id": "ad918393", + "metadata": {}, + "source": [ + "## PySpark" + ] + }, + { + "cell_type": "markdown", + "id": "fd1daec3", + "metadata": {}, + "source": [ + "### Convert numpy dataset to Pandas DataFrame" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "42c5feba", + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "from pyspark import SparkConf\n", + "from pyspark.sql import SparkSession\n", + "from pyspark.sql.types import StructType, StructField, ArrayType, FloatType" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "f063cbe7", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "((10000, 28, 28), dtype('uint8'))" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data = test_data.data.numpy()\n", + "data.shape, data.dtype" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "8c828393", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "((10000, 784), dtype('float64'))" + ] + }, + "execution_count": 23, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "data = data.reshape(10000, 784) / 255.0\n", + "data.shape, data.dtype" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "7760bdbe", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
0123456789...774775776777778779780781782783
00.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
10.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0078430.0117650.00.0117650.6823530.7411760.2627450.00.00.0
20.00.00.00.0000000.00.0000000.0000000.00.0039220.000000...0.6431370.2274510.00.0000000.0000000.0000000.0000000.00.00.0
30.00.00.00.0000000.00.0000000.0000000.00.0000000.082353...0.0039220.0000000.00.0000000.0000000.0000000.0000000.00.00.0
40.00.00.00.0078430.00.0039220.0039220.00.0000000.000000...0.2784310.0470590.00.0000000.0000000.0000000.0000000.00.00.0
..................................................................
99950.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
99960.00.00.00.0000000.00.0000000.0000000.00.0000000.121569...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
99970.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.1058820.0000000.00.0000000.0000000.0000000.0000000.00.00.0
99980.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
99990.00.00.00.0000000.00.0000000.0000000.00.0000000.000000...0.0000000.0000000.00.0000000.0000000.0000000.0000000.00.00.0
\n", + "

10000 rows × 784 columns

\n", + "
" + ], + "text/plain": [ + " 0 1 2 3 4 5 6 7 8 \\\n", + "0 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", + "1 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", + "2 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.003922 \n", + "3 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", + "4 0.0 0.0 0.0 0.007843 0.0 0.003922 0.003922 0.0 0.000000 \n", + "... ... ... ... ... ... ... ... ... ... \n", + "9995 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", + "9996 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", + "9997 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", + "9998 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", + "9999 0.0 0.0 0.0 0.000000 0.0 0.000000 0.000000 0.0 0.000000 \n", + "\n", + " 9 ... 774 775 776 777 778 779 \\\n", + "0 0.000000 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \n", + "1 0.000000 ... 0.007843 0.011765 0.0 0.011765 0.682353 0.741176 \n", + "2 0.000000 ... 0.643137 0.227451 0.0 0.000000 0.000000 0.000000 \n", + "3 0.082353 ... 0.003922 0.000000 0.0 0.000000 0.000000 0.000000 \n", + "4 0.000000 ... 0.278431 0.047059 0.0 0.000000 0.000000 0.000000 \n", + "... ... ... ... ... ... ... ... ... \n", + "9995 0.000000 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \n", + "9996 0.121569 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \n", + "9997 0.000000 ... 0.105882 0.000000 0.0 0.000000 0.000000 0.000000 \n", + "9998 0.000000 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \n", + "9999 0.000000 ... 0.000000 0.000000 0.0 0.000000 0.000000 0.000000 \n", + "\n", + " 780 781 782 783 \n", + "0 0.000000 0.0 0.0 0.0 \n", + "1 0.262745 0.0 0.0 0.0 \n", + "2 0.000000 0.0 0.0 0.0 \n", + "3 0.000000 0.0 0.0 0.0 \n", + "4 0.000000 0.0 0.0 0.0 \n", + "... ... ... ... ... \n", + "9995 0.000000 0.0 0.0 0.0 \n", + "9996 0.000000 0.0 0.0 0.0 \n", + "9997 0.000000 0.0 0.0 0.0 \n", + "9998 0.000000 0.0 0.0 0.0 \n", + "9999 0.000000 0.0 0.0 0.0 \n", + "\n", + "[10000 rows x 784 columns]" + ] + }, + "execution_count": 24, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "pdf784 = pd.DataFrame(data)\n", + "pdf784" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "f7d2bc0d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 87.6 ms, sys: 56.2 ms, total: 144 ms\n", + "Wall time: 143 ms\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
data
0[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
1[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
2[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.003...
3[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
4[0.0, 0.0, 0.0, 0.00784313725490196, 0.0, 0.00...
......
9995[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
9996[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
9997[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
9998[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
9999[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...
\n", + "

10000 rows × 1 columns

\n", + "
" + ], + "text/plain": [ + " data\n", + "0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", + "1 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", + "2 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.003...\n", + "3 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", + "4 [0.0, 0.0, 0.0, 0.00784313725490196, 0.0, 0.00...\n", + "... ...\n", + "9995 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", + "9996 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", + "9997 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", + "9998 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", + "9999 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", + "\n", + "[10000 rows x 1 columns]" + ] + }, + "execution_count": 25, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "%%time\n", + "# 1 column of array\n", + "pdf1 = pd.DataFrame()\n", + "pdf1['data'] = pdf784.values.tolist()\n", + "pdf1" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "a5d7ccf1", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "24/10/08 00:30:18 WARN Utils: Your hostname, cb4ae00-lcedt resolves to a loopback address: 127.0.1.1; using 10.110.47.100 instead (on interface eno1)\n", + "24/10/08 00:30:18 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n", + "Setting default log level to \"WARN\".\n", + "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n", + "24/10/08 00:30:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n" + ] + } + ], + "source": [ + "import os\n", + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" + ] + }, + { + "cell_type": "markdown", + "id": "320760db", + "metadata": {}, + "source": [ + "#### Create Spark DataFrame from Pandas DataFrame" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "4863d5ff", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/pyspark/sql/pandas/serializers.py:224: DeprecationWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, pd.CategoricalDtype) instead\n", + " if is_categorical_dtype(series.dtype):\n", + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 389 ms, sys: 63.2 ms, total: 452 ms\n", + "Wall time: 1.44 s\n" + ] + } + ], + "source": [ + "%%time\n", + "# force FloatType since Spark defaults to DoubleType\n", + "schema = StructType([StructField(\"data\",ArrayType(FloatType()), True)])\n", + "df = spark.createDataFrame(pdf1, schema).repartition(8)" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "406edba5", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "StructType([StructField('data', ArrayType(FloatType(), True), True)])" + ] + }, + "execution_count": 28, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df.schema" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "831f4a01-3a49-4114-b9a0-2ae54526d72d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 61 ms, sys: 21.6 ms, total: 82.6 ms\n", + "Wall time: 854 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "# force FloatType since Spark defaults to DoubleType\n", + "schema = StructType([StructField(str(x), FloatType(), True) for x in range(784)])\n", + "df784 = spark.createDataFrame(pdf784, schema).repartition(8)" + ] + }, + { + "cell_type": "markdown", + "id": "ac4c7448", + "metadata": {}, + "source": [ + "### Save the test dataset as parquet files" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "e8ebae46", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "24/10/08 00:30:21 WARN TaskSetManager: Stage 0 contains a task of very large size (4032 KiB). The maximum recommended task size is 1000 KiB.\n", + "[Stage 0:> (0 + 8) / 8]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 2.09 ms, sys: 2.57 ms, total: 4.66 ms\n", + "Wall time: 1.78 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "df.write.mode(\"overwrite\").parquet(\"fashion_mnist_1\")" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "922314ce-2996-4666-9fc9-bcd98d16bb56", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "24/10/08 00:30:23 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.\n", + "24/10/08 00:30:23 WARN TaskSetManager: Stage 3 contains a task of very large size (7849 KiB). The maximum recommended task size is 1000 KiB.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 3.01 ms, sys: 22 μs, total: 3.03 ms\n", + "Wall time: 734 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "df784.write.mode(\"overwrite\").parquet(\"fashion_mnist_784\")" + ] + }, + { + "cell_type": "markdown", + "id": "8688429e", + "metadata": {}, + "source": [ + "### Check arrow memory configuration" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "088cb37f", + "metadata": {}, + "outputs": [], + "source": [ + "spark.conf.set(\"spark.sql.execution.arrow.maxRecordsPerBatch\", \"128\")\n", + "# This line will fail if the vectorized reader runs out of memory\n", + "assert len(df.head()) > 0, \"`df` should not be empty\"" + ] + }, + { + "cell_type": "markdown", + "id": "d7c77eb4-7bd6-40c7-9a35-ee899a66ece3", + "metadata": {}, + "source": [ + "## Inference using Spark DL API" + ] + }, + { + "cell_type": "markdown", + "id": "59395856-a588-43c6-93c8-c83100716ac1", + "metadata": { + "tags": [] + }, + "source": [ + "### 1 columns of 784 float" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "133cc9a5-64c6-4820-807e-b87cf7e0b75a", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "import numpy as np\n", + "\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import struct, col, array\n", + "from pyspark.sql.types import ArrayType, FloatType, Union, Dict" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "79b151d9-d112-43b6-a479-887e2fd0e2b1", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "1" + ] + }, + "execution_count": 34, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = spark.read.parquet(\"fashion_mnist_1\")\n", + "len(df.columns)" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "cabcd546-2e8e-40d0-8b79-7598a7a83aae", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "StructType([StructField('data', ArrayType(FloatType(), True), True)])" + ] + }, + "execution_count": 35, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df.schema" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "823c3825", + "metadata": {}, + "outputs": [], + "source": [ + "# Get absolute path to model\n", + "model_path = \"{}/model.pt\".format(os.getcwd())" + ] + }, + { + "cell_type": "markdown", + "id": "2d7c2bac", + "metadata": {}, + "source": [ + "For inference on Spark, we'll compile the model with the Torch-TensorRT AOT compiler and cache on the executor. We can specify dynamic batch sizes before compilation to [optimize across multiple input shapes](https://pytorch.org/TensorRT/user_guide/dynamic_shapes.html)." + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "73dc73cb-25e3-4798-a019-e1abd684eaa1", + "metadata": {}, + "outputs": [], + "source": [ + "def predict_batch_fn():\n", + " import torch\n", + " import torch_tensorrt as trt\n", + " \n", + " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + " if device != \"cuda\":\n", + " raise ValueError(\"This function uses the TensorRT model which requires a GPU device\")\n", + "\n", + " # Define model\n", + " class NeuralNetwork(nn.Module):\n", + " def __init__(self):\n", + " super(NeuralNetwork, self).__init__()\n", + " self.linear_relu_stack = nn.Sequential(\n", + " nn.Linear(28*28, 512),\n", + " nn.ReLU(),\n", + " nn.Linear(512, 512),\n", + " nn.ReLU(),\n", + " nn.Linear(512, 10)\n", + " )\n", + "\n", + " def forward(self, x):\n", + " logits = self.linear_relu_stack(x)\n", + " return logits\n", + "\n", + " model = NeuralNetwork().to(device)\n", + " model.load_state_dict(torch.load(model_path, weights_only=True))\n", + "\n", + " # Preparing the inputs for dynamic batch sizing.\n", + " inputs = [trt.Input(min_shape=(1, 784), \n", + " opt_shape=(50, 784), \n", + " max_shape=(64, 784), \n", + " dtype=torch.float32)]\n", + "\n", + " # Trace the computation graph and compile to produce an optimized module\n", + " trt_gm = trt.compile(model, ir=\"dynamo\", inputs=inputs, require_full_compilation=True)\n", + "\n", + " def predict(inputs: np.ndarray):\n", + " print(\"Predicting on process PID: {}\".format(os.getpid()))\n", + " stream = torch.cuda.Stream()\n", + " with torch.no_grad(), torch.cuda.stream(stream):\n", + " # use array to combine columns into tensors\n", + " torch_inputs = torch.from_numpy(inputs).to(device)\n", + " outputs = trt_gm(torch_inputs)\n", + " return outputs.detach().cpu().numpy()\n", + " \n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "df68cca1-2d47-4e88-8aad-9899402aee97", + "metadata": {}, + "outputs": [], + "source": [ + "mnist = predict_batch_udf(predict_batch_fn,\n", + " input_tensor_shapes=[[784]],\n", + " return_type=ArrayType(FloatType()),\n", + " batch_size=50)" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "63555b3b-3673-4712-97aa-fd728c6c4979", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 189 ms, sys: 53.6 ms, total: 242 ms\n", + "Wall time: 9.73 s\n" + ] + } + ], + "source": [ + "%%time\n", + "# first pass caches model/fn\n", + "preds = df.withColumn(\"preds\", mnist(struct(df.columns))).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "5dbf058a-70d6-4199-af9d-13843d078950", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 195 ms, sys: 75 ms, total: 270 ms\n", + "Wall time: 1.26 s\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(*df.columns)).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "id": "3f5ed801-6ca5-43a0-bf9c-2535a0dfe2e8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 199 ms, sys: 54.9 ms, total: 254 ms\n", + "Wall time: 1.27 s\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(*[col(c) for c in df.columns])).collect()" + ] + }, + { + "cell_type": "markdown", + "id": "c6dbec03-9b64-46c4-a748-f889be571384", + "metadata": { + "tags": [] + }, + "source": [ + "### Check predictions" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "id": "f1f1e5fd-5866-4b78-b9d3-709e6b383a0c", + "metadata": {}, + "outputs": [], + "source": [ + "predictions = preds[0].preds\n", + "img = preds[0].data" + ] + }, + { + "cell_type": "code", + "execution_count": 44, + "id": "76b76502-adb7-45ec-a365-2e61cdd576fc", + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "id": "c163953a-1504-444f-b39f-86b61d34e440", + "metadata": {}, + "outputs": [], + "source": [ + "img = np.array(img).reshape(28,28)" + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "id": "bc0fad05-50ab-4ae5-b9fd-e50133c4c92a", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAknklEQVR4nO3de3SV9Z3v8c9OSDa3ZMcQcpNAAyi0cumUSsqoFEsOkM64QDkdb3MOuDow0uCqUqsnPVZq27PS4hrrqUNxrbNaqKvihXNERsdiBSWMCnRAGMZeUsAoYSChYJMNCUl2sn/nD8bMREH4/kzyS8L7tdZei+z9fHh+efIknzzZO99EnHNOAAD0spTQCwAAXJooIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBDAq9gA9LJpM6evSoMjIyFIlEQi8HAGDknNOpU6dUWFiolJTzX+f0uQI6evSoioqKQi8DAPAJ1dbWatSoUed9vM8VUEZGhiTpWn1Zg5QWeDXodr11VcuEKSCYdiX0ul7q/Hp+Pj1WQKtXr9bDDz+suro6TZ06VY899pimT59+wdwHP3YbpDQNilBAA06v/ViVAgKC+fdPvws9jdIjL0J45plntGLFCq1cuVJvvfWWpk6dqrlz5+r48eM9sTsAQD/UIwX0yCOPaMmSJbrjjjv0mc98Ro8//riGDh2qn/3sZz2xOwBAP9TtBdTW1qY9e/aotLT0P3aSkqLS0lLt2LHjI9u3trYqHo93uQEABr5uL6ATJ06oo6NDeXl5Xe7Py8tTXV3dR7avrKxULBbrvPEKOAC4NAT/RdSKigo1NjZ23mpra0MvCQDQC7r9VXA5OTlKTU1VfX19l/vr6+uVn5//ke2j0aii0Wh3LwMA0Md1+xVQenq6pk2bpq1bt3bel0wmtXXrVs2YMaO7dwcA6Kd65PeAVqxYoUWLFunzn/+8pk+frkcffVRNTU264447emJ3AIB+qEcK6Oabb9Yf//hHPfjgg6qrq9NnP/tZbd68+SMvTAAAXLoizvWtmSXxeFyxWEyzNJ9JCBiwUscXmzPH/s7+XGnu/N+bM8An1e4S2qZNamxsVGZm5nm3C/4qOADApYkCAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQfTINGygv6r5gf1vVt0/f6M5U9Vw/gGN5zM+7Yw5k7/fnpGkX2yYbc4Ufe9Nr33h0sUVEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIJgGnYfFhlk//C4jg77jpyzZzxF0tLNGZdoM2cGFY8xZyRpy20PmzNf/NXd5syVf7PbnKk3J6TdN1/vkZIe/O5T5sza7/kdc7NIxJ7pxXMcF48rIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAIIuJc35rSF4/HFYvFNEvzNSiSFno53cdjgGIkNdWc8RpG6qtvnTpd/GHtNK/c2KI/mjODSg977asvO/NysTlzw+X7zZktkzLMGfR97S6hbdqkxsZGZWZmnnc7roAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIIhBoRdwyfAY3Ona2+378Rh62peHikpSytRPmzNbvvS/vfZV+ssV5syV8hhGmmIfNBtJsX9svc4hSWk/zDZnbl73L+bMhsXfNGcuW7fDnEHfxBUQACAICggAEES3F9B3vvMdRSKRLreJEyd2924AAP1cjzwHdNVVV2nLli3/sZNBPNUEAOiqR5ph0KBBys/P74n/GgAwQPTIc0AHDhxQYWGhxo4dq9tvv12HD5//VUKtra2Kx+NdbgCAga/bC6ikpETr1q3T5s2btWbNGtXU1Oi6667TqVOnzrl9ZWWlYrFY562oqKi7lwQA6IO6vYDKysr0la98RVOmTNHcuXP10ksvqaGhQc8+++w5t6+oqFBjY2Pnrba2truXBADog3r81QFZWVm68sordfDgwXM+Ho1GFY1Ge3oZAIA+psd/D+j06dM6dOiQCgoKenpXAIB+pNsL6N5771VVVZXeffddvfnmm7rxxhuVmpqqW2+9tbt3BQDox7r9R3BHjhzRrbfeqpMnT2rkyJG69tprtXPnTo0cObK7dwUA6Me6vYCefvrp7v4vYeEzWNRjMKYkKdlhjsRv/YI5M2b5H8yZx09eZ85IUuGrvTSdyiXNkUh0qH03nsNI675gf172nUSmOfPMQw+bMyv/9svmTP0Mfr2jL2IWHAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAE0eN/kK7XRCL2jM/gzt7kMyTUY8ilz1BRX1d/Y485U92YZ840t6ebM5I0/NmdXjmrSKrnANheknbKnnmz6Qpz5tE/fcqcuWvUFnPmgduWmDOSlLne43zora9FPvvx3VcP4QoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQQycadh9ncfkWp+JyS7Re5Oth20fac60O/uY5dQU+4TvdzeNNWckqUB1Xjkr1+HxcWpLdP9CziPvsTfNmW9VVJsz1x69ypxZeWC+OXPz/9xszkjSyy8UmTPJUx6jxH34TrXuQ385gCsgAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAhi4Awj9RiWF0lL99tVos0jZF+f1348HL3vz71y38x91px58t++YM4kZR+eWPCIfZhmr+rD54OvXzWnmTP/fcxOc2bV3jnmTGqR3zDNjF/av0Y0Xuu1q94T8bjucD0z5JgrIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAIIuKcx1TEHhSPxxWLxTRL8zUoYh9uOJAcX24fEtqaZd/Pr5ausockrWv4vDkzKv19c+Z7v7zJnIn9wT7AVJKu/5td5szG33zWnBn0b1FzJqXN432K+H16++yrdUTSnBk54YQ5E4u2mDNn2v2+lqwc/w/mzLINS82Z4v+xw5zpy9pdQtu0SY2NjcrMzDzvdlwBAQCCoIAAAEGYC2j79u264YYbVFhYqEgkoueff77L4845PfjggyooKNCQIUNUWlqqAwcOdNd6AQADhLmAmpqaNHXqVK1evfqcj69atUo//vGP9fjjj2vXrl0aNmyY5s6dq5YW+89tAQADl/kvopaVlamsrOycjznn9Oijj+qBBx7Q/PnzJUlPPPGE8vLy9Pzzz+uWW275ZKsFAAwY3focUE1Njerq6lRaWtp5XywWU0lJiXbsOPerPFpbWxWPx7vcAAADX7cWUF1dnSQpLy+vy/15eXmdj31YZWWlYrFY562oqKg7lwQA6KOCvwquoqJCjY2Nnbfa2trQSwIA9IJuLaD8/HxJUn19fZf76+vrOx/7sGg0qszMzC43AMDA160FVFxcrPz8fG3durXzvng8rl27dmnGjBnduSsAQD9nfhXc6dOndfDgwc63a2pqtG/fPmVnZ2v06NG6++679f3vf19XXHGFiouL9e1vf1uFhYVasGBBd64bANDPmQto9+7duv766zvfXrFihSRp0aJFWrdune677z41NTVp6dKlamho0LXXXqvNmzdr8ODB3bdqAEC/d0kPI31nld+PBf/uxp+bMyv++a/MmfT0dnPmB1OfM2c2N0wxZyQpRfZT53eNeRfe6EPe23u5OdNxWcKckaTBtenmTOY79uOQ0m7PdKTbB4Qmzd9inuVS7Zlkmn19kaT9OJye2WzOfLboiDkjSUdPx8yZa/LeMWfeet/+6t8j72eZM5I0+iv/6pWzYBgpAKBPo4AAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAjPWbkDw29v/3uv3MKDf2HO5Pyj/c9RnF54ypz52dHrzJnGNr8/lXFH0RvmzIm2YeZMzeCkOaMO+2RmSWq7zL6vxFf+ZM6MijWaMyOjp82ZaKp9orokZQ2yT5xOeIzQbuqImjMzM6vt+0na9yNJbw4ab86kyn4ODRvUZs68NH2NOSNJt95+rzkTe3Kn174uhCsgAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAhiwAwjbblhujmTFtnnt69v5ZszOf/rPXPm5fHPmTMPn7Afh6Ep9kGIkrTy9QXmTErcfsq5LI+Bmh7zS8/uK2HOxA9cZs4cbBhhztTaZ54qtdXZQ546ovYBsM5jZuzr6Z8zZ25f/Ip9R5Kuy/qDOfO5wYfNmZfTrjJn/uKf7zRnJOlvKn5lzrz8ZKbXvi6EKyAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACGLADCOt/3zvvStZP6w1Z/4y51/MmZ822AcU/lXWP5sz3639S3NGki7/Zao5kxjqMX1SaeZEJOk3hNOl2NfXkW7fTzLNvj6ftbVm+RxvSR6xiMfMWJ/9DP83+6TZx//pevuOJP1h/hpz5jdt9ndqUWy/OfOPmZPNGUn669i/mjO/+rOlpu0jHa3Sv2y64HZcAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEANmGGna6d7b172XbzZnNvxpujmTmx43Z7556L+aM81/f7k5I0lncu3fv/gMx0xtM0eUTPUbwpnSSwM1fQZ3OvvsV6+1SVL7EL+clc9xODXaft4VVNn3I0lfmzbTnCkc3GDOVJ/OM2cWXr7XnJGkESn2D+6Zy4eZtm9PpEoXMX+ZKyAAQBAUEAAgCHMBbd++XTfccIMKCwsViUT0/PPPd3l88eLFikQiXW7z5s3rrvUCAAYIcwE1NTVp6tSpWr169Xm3mTdvno4dO9Z5e+qppz7RIgEAA4/5RQhlZWUqKyv72G2i0ajy8/O9FwUAGPh65Dmgbdu2KTc3VxMmTNCyZct08uTJ827b2tqqeDze5QYAGPi6vYDmzZunJ554Qlu3btUPf/hDVVVVqaysTB0dHefcvrKyUrFYrPNWVFTU3UsCAPRB3f57QLfcckvnvydPnqwpU6Zo3Lhx2rZtm2bPnv2R7SsqKrRixYrOt+PxOCUEAJeAHn8Z9tixY5WTk6ODBw+e8/FoNKrMzMwuNwDAwNfjBXTkyBGdPHlSBQUFPb0rAEA/Yv4R3OnTp7tczdTU1Gjfvn3Kzs5Wdna2HnroIS1cuFD5+fk6dOiQ7rvvPo0fP15z587t1oUDAPo3cwHt3r1b119/fefbHzx/s2jRIq1Zs0b79+/Xz3/+czU0NKiwsFBz5szR9773PUWj0e5bNQCg3zMX0KxZs+ScO+/jL7/88idakK9Bzb23rwNt9t9xejjfPjjw/522Px/W+LP/Ys4kY34TKxPD7LnIuV8M+bE60u0Z3yGcH3Nqn39XvTRY1GtAqOdxGHTGnknxGBrrcz74HLuOqN+B+PX6qebMhhUPmzP/lD7OnLl6yLvmjCTFk0lzZlj1CdP27R2tF7Uds+AAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQRLf/Se5QhpywT3j1NS7tuDlzpN0+Xvi+l+40ZzJG2L+nSAwzRyRJ6XF7xnl8y9Mx2J6Rx1RrSWof6rErj+nMPuvzmbrtKzHcvsCkx1eT1Db7lOqUixu03EX8U37TsDMO24/Dtw7PN2f+77gt5sy2Mx4nq6TC1FPmTMeBd2zbu8RFbccVEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEMWCGkWYeOm3OJFyH177GprWYMzN3LDNn8t+wD0L800RzxHvIZUuOPdMRtb9P0fc9Bkl6fmvlMyy1faj9fWrPsJ97KcMvbsDjf5Zs9ZmUKkWPppkzaaftHyevY+cxnDa11W8Y6Zlce+6d9VeYM7+7/x/MGWm4R0a6LGWIOZM6Ybxpe9fRKh248HZcAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEANmGGlKc5s5kxbxG9T4r22Z5szwX9kHB/5poscAxaQ90jHEPhBSktqy7DuLnrAf8/RT9vWlLzhuzkjSycZh5kxHm/3TKP1w1JzJ3m7/fjHi96HVmWz7uddc6DFY1GMYqXzmivrNIlUiw76+ISn2j9P8nXeaM7+c8RNzRpLaZT/3IgnbxOJI8uK25woIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIIYMMNI1d5hjjQmz3jt6m93fs2cSRtln4bYmmN/n1Ja7fuJJPwmNabnN5szY38UN2fa3ztiztRHS8wZSRq/9aQ9dLTWnskdYY68d1OuOdM8xjZE8gORofbhvu6Mx3DfpMf52m7PdKT6TWV1g+y55lH2TMZO+7DiuulDzRlJGpdmv+5of+dd2/YucVHbcQUEAAiCAgIABGEqoMrKSl199dXKyMhQbm6uFixYoOrq6i7btLS0qLy8XCNGjNDw4cO1cOFC1dfXd+uiAQD9n6mAqqqqVF5erp07d+qVV15RIpHQnDlz1NTU1LnNPffcoxdeeEEbNmxQVVWVjh49qptuuqnbFw4A6N9ML0LYvHlzl7fXrVun3Nxc7dmzRzNnzlRjY6N++tOfav369frSl74kSVq7dq0+/elPa+fOnfrCF77QfSsHAPRrn+g5oMbGRklSdna2JGnPnj1KJBIqLS3t3GbixIkaPXq0duzYcc7/o7W1VfF4vMsNADDweRdQMpnU3XffrWuuuUaTJk2SJNXV1Sk9PV1ZWVldts3Ly1NdXd05/5/KykrFYrHOW1FRke+SAAD9iHcBlZeX6+2339bTTz/9iRZQUVGhxsbGzlttrcfvVAAA+h2vX0Rdvny5XnzxRW3fvl2jRo3qvD8/P19tbW1qaGjochVUX1+v/Pz8c/5f0WhU0WjUZxkAgH7MdAXknNPy5cu1ceNGvfrqqyouLu7y+LRp05SWlqatW7d23lddXa3Dhw9rxowZ3bNiAMCAYLoCKi8v1/r167Vp0yZlZGR0Pq8Ti8U0ZMgQxWIxffWrX9WKFSuUnZ2tzMxM3XXXXZoxYwavgAMAdGEqoDVr1kiSZs2a1eX+tWvXavHixZKkH/3oR0pJSdHChQvV2tqquXPn6ic/+Um3LBYAMHCYCsi5Cw/ZGzx4sFavXq3Vq1d7L8pLqv31FC82jbrwRufgkh4Zj3mfqc0eQwOHewwwjfi9FiXZan8KsfnKkeZMes175szlzx4yZySp/i/GmjMnr8kwZ7JGnDZnWk/bh+dG3k83ZyRJDWn2ffnMtPU59XwyfrNIFUnYd+aG2wfANhfa9/PXW/7WnJGkmr/8P+ZM6ohs0/Yu2Sa9f+HtmAUHAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAILz+Impf1PG7A+bMsJRWr309cc1PzZm/Tiw1ZyLNqeZMaixhziQ7/CYmuxb76RNf3mjOpNx1pTnT3Gqf5ixJzp2yh94fYo40vptlzkTsg86lNM8x0PZTz2vitEv1CHlMo4/4jKOX5AbbD3pqg/3zomOY/Z1Kr++9L98tf1Z84Y3+k/b2Fum1C2/HFRAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABDFghpH6yEpp9sqlekxd/NF1T5szhYP+ZM784uSfmzMvvjHNnJEkddgHPL5/JMucSW22f5/kevFbq4jHQE2XZh8+6fxmxnqJtHsM7/SZ9+kzK9VjPy7FcyirxzmejHruyygl4TdgtTnZZs60Zdmqoj1xcdtzBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVzSw0i3nLrKKzdl6GFz5nKPwaIT0trNmU8NPmnO/LdZ/2TOSNL631xtDx0eYo6k2A+DEjH7sE9JingMn4wk7ZmUM36DJK1c7+xGkhTxmMHpUuwL9Bk067O2s/vyead8ziH7bga12DOSVNPeYc4MPpEwbd/efnHbcwUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFc2sNIj07wysVGN5sz49L+aM78In6lOZMWsQ8aLMvYb85I0pTP15ozI0pOmzN5qfbMxvifmTOS9NJR+4Da9qT9+7jmtjRzJumxn2iabYjkB4Z4DML1OQ6DUuxTOH3mip5uiXqkpNgQ+8TPwuGN5kxbR6o5k/SZyirpjTPjzJljMwabtu9olXQRM465AgIABEEBAQCCMBVQZWWlrr76amVkZCg3N1cLFixQdXV1l21mzZqlSCTS5XbnnXd266IBAP2fqYCqqqpUXl6unTt36pVXXlEikdCcOXPU1NTUZbslS5bo2LFjnbdVq1Z166IBAP2f6UUImzdv7vL2unXrlJubqz179mjmzJmd9w8dOlT5+fnds0IAwID0iZ4Damw8+2qP7OzsLvc/+eSTysnJ0aRJk1RRUaHm5vO/aqy1tVXxeLzLDQAw8Hm/DDuZTOruu+/WNddco0mTJnXef9ttt2nMmDEqLCzU/v37df/996u6ulrPPffcOf+fyspKPfTQQ77LAAD0U94FVF5errfffluvv/56l/uXLl3a+e/JkyeroKBAs2fP1qFDhzRu3Edff15RUaEVK1Z0vh2Px1VUVOS7LABAP+FVQMuXL9eLL76o7du3a9SoUR+7bUlJiSTp4MGD5yygaDSqaNTvl8QAAP2XqYCcc7rrrru0ceNGbdu2TcXFxRfM7Nu3T5JUUFDgtUAAwMBkKqDy8nKtX79emzZtUkZGhurq6iRJsVhMQ4YM0aFDh7R+/Xp9+ctf1ogRI7R//37dc889mjlzpqZMmdIj7wAAoH8yFdCaNWsknf1l0/9s7dq1Wrx4sdLT07VlyxY9+uijampqUlFRkRYuXKgHHnig2xYMABgYzD+C+zhFRUWqqqr6RAsCAFwaLulp2EM8JwUvy/qNOXMwETFnyrPs06b92CfxnmX/na3mZJs5MzRlqDnz6ZzqC290Dssu22vOXJZqX5+PN1rsk6Mbkn5ry0+1f2wHe0xi75D986LF2c/XkSmt5owkFacNN2f2tNrP8fFp9mO3oyXLnJGk9X8sMWdGVb5p2r7dJXTgIrZjGCkAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABHFpDyP9ut9fYi2b8HVzJtpgH3yaTOud7w86on77OXK9PRfJsw+FzHhziDlT+Muj5owkuVT7+9Rx2TBzJvVUizmjY8fNEZdot+9HUmSofYhpZLjH4NMLTNg/p3b74E5fzVfZ/5Cmz+ft8D2HzZn2Y3XmzFn2QbM9hSsgAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQRJ+bBef+fTZUuxKSx5go07467HPJJKk9YZ/jldruMQsu0kuz4FL89pNs8ZgF12w/5h1tEXOmPen3sXUe35N1tKfa9+Nz7rk2e8R5zoJL2r80RJL24+A1Cy7Ze7Pg2tvtn+tJj3OoPWn/2LY7+9eU3tKus2tzF/j4RtyFtuhlR44cUVFRUehlAAA+odraWo0aNeq8j/e5Akomkzp69KgyMjIUiXT9zjcej6uoqEi1tbXKzMwMtMLwOA5ncRzO4jicxXE4qy8cB+ecTp06pcLCQqV8zE9Y+tyP4FJSUj62MSUpMzPzkj7BPsBxOIvjcBbH4SyOw1mhj0MsFrvgNrwIAQAQBAUEAAiiXxVQNBrVypUrFY36/SXTgYLjcBbH4SyOw1kch7P603Hocy9CAABcGvrVFRAAYOCggAAAQVBAAIAgKCAAQBD9poBWr16tT33qUxo8eLBKSkr061//OvSSet13vvMdRSKRLreJEyeGXlaP2759u2644QYVFhYqEono+eef7/K4c04PPvigCgoKNGTIEJWWlurAgQNhFtuDLnQcFi9e/JHzY968eWEW20MqKyt19dVXKyMjQ7m5uVqwYIGqq6u7bNPS0qLy8nKNGDFCw4cP18KFC1VfXx9oxT3jYo7DrFmzPnI+3HnnnYFWfG79ooCeeeYZrVixQitXrtRbb72lqVOnau7cuTp+/HjopfW6q666SseOHeu8vf7666GX1OOampo0depUrV69+pyPr1q1Sj/+8Y/1+OOPa9euXRo2bJjmzp2rlhb7IMm+7ELHQZLmzZvX5fx46qmnenGFPa+qqkrl5eXauXOnXnnlFSUSCc2ZM0dNTU2d29xzzz164YUXtGHDBlVVVeno0aO66aabAq66+13McZCkJUuWdDkfVq1aFWjF5+H6genTp7vy8vLOtzs6OlxhYaGrrKwMuKret3LlSjd16tTQywhKktu4cWPn28lk0uXn57uHH364876GhgYXjUbdU089FWCFvePDx8E55xYtWuTmz58fZD2hHD9+3ElyVVVVzrmzH/u0tDS3YcOGzm1+97vfOUlux44doZbZ4z58HJxz7otf/KL7+te/Hm5RF6HPXwG1tbVpz549Ki0t7bwvJSVFpaWl2rFjR8CVhXHgwAEVFhZq7Nixuv3223X48OHQSwqqpqZGdXV1Xc6PWCymkpKSS/L82LZtm3JzczVhwgQtW7ZMJ0+eDL2kHtXY2ChJys7OliTt2bNHiUSiy/kwceJEjR49ekCfDx8+Dh948sknlZOTo0mTJqmiokLNzc0hlndefW4Y6YedOHFCHR0dysvL63J/Xl6efv/73wdaVRglJSVat26dJkyYoGPHjumhhx7Sddddp7ffflsZGRmhlxdEXV2dJJ3z/PjgsUvFvHnzdNNNN6m4uFiHDh3St771LZWVlWnHjh1KTfX4Wz19XDKZ1N13361rrrlGkyZNknT2fEhPT1dWVlaXbQfy+XCu4yBJt912m8aMGaPCwkLt379f999/v6qrq/Xcc88FXG1Xfb6A8B/Kyso6/z1lyhSVlJRozJgxevbZZ/XVr3414MrQF9xyyy2d/548ebKmTJmicePGadu2bZo9e3bAlfWM8vJyvf3225fE86Af53zHYenSpZ3/njx5sgoKCjR79mwdOnRI48aN6+1lnlOf/xFcTk6OUlNTP/Iqlvr6euXn5wdaVd+QlZWlK6+8UgcPHgy9lGA+OAc4Pz5q7NixysnJGZDnx/Lly/Xiiy/qtdde6/LnW/Lz89XW1qaGhoYu2w/U8+F8x+FcSkpKJKlPnQ99voDS09M1bdo0bd26tfO+ZDKprVu3asaMGQFXFt7p06d16NAhFRQUhF5KMMXFxcrPz+9yfsTjce3ateuSPz+OHDmikydPDqjzwzmn5cuXa+PGjXr11VdVXFzc5fFp06YpLS2ty/lQXV2tw4cPD6jz4ULH4Vz27dsnSX3rfAj9KoiL8fTTT7toNOrWrVvnfvvb37qlS5e6rKwsV1dXF3ppveob3/iG27Ztm6upqXFvvPGGKy0tdTk5Oe748eOhl9ajTp065fbu3ev27t3rJLlHHnnE7d2717333nvOOed+8IMfuKysLLdp0ya3f/9+N3/+fFdcXOzOnDkTeOXd6+OOw6lTp9y9997rduzY4WpqatyWLVvc5z73OXfFFVe4lpaW0EvvNsuWLXOxWMxt27bNHTt2rPPW3Nzcuc2dd97pRo8e7V599VW3e/duN2PGDDdjxoyAq+5+FzoOBw8edN/97nfd7t27XU1Njdu0aZMbO3asmzlzZuCVd9UvCsg55x577DE3evRol56e7qZPn+527twZekm97uabb3YFBQUuPT3dXX755e7mm292Bw8eDL2sHvfaa685SR+5LVq0yDl39qXY3/72t11eXp6LRqNu9uzZrrq6Ouyie8DHHYfm5mY3Z84cN3LkSJeWlubGjBnjlixZMuC+STvX+y/JrV27tnObM2fOuK997Wvusssuc0OHDnU33nijO3bsWLhF94ALHYfDhw+7mTNnuuzsbBeNRt348ePdN7/5TdfY2Bh24R/Cn2MAAATR558DAgAMTBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAI4v8DIE1CeiobCX8AAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.figure()\n", + "plt.imshow(img)\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "id": "56f36efb-e3a2-49f9-b9fb-1657bc25e5c5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[-1.309907078742981, -3.8460376262664795, 0.845407247543335, -2.5534284114837646, 0.7116107940673828, 0.9341840147972107, 0.4921048879623413, 0.22850888967514038, 2.951157331466675, 1.0042279958724976]\n", + "predicted label: Bag\n" + ] + } + ], + "source": [ + "print(predictions)\n", + "print(\"predicted label:\", classes[np.argmax(predictions)])" + ] + }, + { + "cell_type": "markdown", + "id": "56ca1195-ea0f-405f-87fe-857e5c0c76a5", + "metadata": {}, + "source": [ + "### 784 columns of float" + ] + }, + { + "cell_type": "code", + "execution_count": 48, + "id": "e0ab0af6-b5c9-4b74-9dd6-baa7737cc986", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "784" + ] + }, + "execution_count": 48, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = spark.read.parquet(\"fashion_mnist_784\")\n", + "len(df.columns)" + ] + }, + { + "cell_type": "code", + "execution_count": 49, + "id": "13ae45dc-85a0-4864-8a58-9dc29ae4efd7", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 266 ms, sys: 69.9 ms, total: 336 ms\n", + "Wall time: 4.05 s\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(struct(df.columns))).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 50, + "id": "0b3fb48b-f871-41f2-ac57-346899a6fe48", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 271 ms, sys: 66.2 ms, total: 337 ms\n", + "Wall time: 1.96 s\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(array(*df.columns))).collect()" + ] + }, + { + "cell_type": "markdown", + "id": "dc48ec42-0df6-4e6a-b019-1270ab71d2cf", + "metadata": { + "tags": [] + }, + "source": [ + "### Check predictions" + ] + }, + { + "cell_type": "code", + "execution_count": 51, + "id": "d815c701-9f5b-422c-b3f9-fbc30456953c", + "metadata": {}, + "outputs": [], + "source": [ + "preds = df.withColumn(\"preds\", mnist(array(*df.columns))).limit(10).toPandas()" + ] + }, + { + "cell_type": "code", + "execution_count": 52, + "id": "b571b742-5079-42b2-8524-9181a0dec2c7", + "metadata": {}, + "outputs": [], + "source": [ + "sample = preds.iloc[0]\n", + "predictions = sample.preds\n", + "img = sample.drop('preds').to_numpy(dtype=float)" + ] + }, + { + "cell_type": "code", + "execution_count": 53, + "id": "d33d6a4e-e6b9-489d-ac21-c4eddc801784", + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np" + ] + }, + { + "cell_type": "code", + "execution_count": 54, + "id": "6d10061e-aca6-4f81-bdfe-72e327ed7349", + "metadata": {}, + "outputs": [], + "source": [ + "img = np.array(img).reshape(28,28)" + ] + }, + { + "cell_type": "code", + "execution_count": 55, + "id": "01f70e08-2c1d-419f-8676-3f6f4aba760f", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAelElEQVR4nO3dfXCU9b338c9mk2wChI0h5KkEGlChlYeeUkm5VYolA6RzPKJMx6c/wHFgtMEpUquTjorazqTFGevoUDx/tFDvW3yaERg9vekomjC2gQ4oN4fTNkJOLHhCgtLmgYQ8kP2dPzhu74UA/V1s8t2E92vmmsnuXt9cX669yGev7LXfhJxzTgAADLM06wYAAFcmAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAm0q0bOFcsFlNzc7NycnIUCoWs2wEAeHLOqbOzUyUlJUpLu/B5TsoFUHNzs0pLS63bAABcpmPHjmnSpEkXfDzlAignJ0eSdKO+o3RlGHeDi0mf/CXvmrL/fcK7Zvd/TfOuyco4410jSWcGhue30gNueM7uY7Fg/56M8IB3TUd7tnfN3KlHvWva/7nbu8b19nrXILgz6tcH+k385/mFDFkAbdy4Uc8884xaWlo0Z84cvfDCC5o3b94l6774tVu6MpQeIoBSWXpaxLsmc5z/cxoe47+dcEbYu0aS3DAFkIYpgEIBAygcIIDS+rK8azLGZnrXpIf6vWtcKOZdg8vwPxNGL/U2ypD8b3vttde0bt06rV+/Xh9++KHmzJmjJUuW6MQJ/1e/AIDRaUgC6Nlnn9WqVat077336qtf/apefPFFjRkzRr/61a+GYnMAgBEo6QHU19en/fv3q6Ki4u8bSUtTRUWF6uvrz1u/t7dXHR0dCQsAYPRLegB9/vnnGhgYUGFhYcL9hYWFamlpOW/9mpoaRaPR+MIVcABwZTD/IGp1dbXa29vjy7Fjx6xbAgAMg6RfBZefn69wOKzW1taE+1tbW1VUVHTe+pFIRJGI/1VOAICRLelnQJmZmZo7d6527doVvy8Wi2nXrl2aP39+sjcHABihhuRzQOvWrdOKFSv0jW98Q/PmzdNzzz2nrq4u3XvvvUOxOQDACDQkAXTHHXfos88+0xNPPKGWlhZ97Wtf086dO8+7MAEAcOUaskkIa9as0Zo1a4bq2yMF/PUG/1E8bxa/6V3zTGand01Z5DPvGkkKy/8T82kBPmU/Ns1/NMyA8/+NeTjgBID/1z3Fu2bP38q8a/51yr951/zL4u9712S99QfvGgw986vgAABXJgIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACaGbBgpRr/2qf6vX/b1hr1rmntzvWvS5LxrJCmmkHdNTyzDuyaa3u1dkxU6410TZFCqJDV2T/SuOdaW613z2+7z/0jlpbRd7f9jy38rGA6cAQEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATDANG4H1FA1410zPOO1dUxJp867Jz+j0rpGkE/3jvWsyQv77oT/m/1+v20W8a3LCPd41klSc1e5d840i/wnkszKPe9f05XiXIEVxBgQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEw0gRWGZBt3dNj/MfWDkuwEDNfhf2rpGkMWl93jWdA1neNe0D2d41n/eO8665Kfdj7xop2D4Ph2IBavyPh1jEvwapiTMgAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJhhGisDGZvd61zQPRLxrYs7/dVK/d8VZGaGBYanpjfn/1ysb87l3TdvAGO+aoE6d8X9um8/keNf0FQd9dpFqOAMCAJgggAAAJpIeQE8++aRCoVDCMmPGjGRvBgAwwg3Je0DXXXed3n333b9vJJ23mgAAiYYkGdLT01VUVDQU3xoAMEoMyXtAhw8fVklJiaZOnap77rlHR48eveC6vb296ujoSFgAAKNf0gOovLxcW7Zs0c6dO7Vp0yY1NTXppptuUmdn56Dr19TUKBqNxpfS0tJktwQASEFJD6DKykp997vf1ezZs7VkyRL95je/UVtbm15//fVB16+urlZ7e3t8OXbsWLJbAgCkoCG/OiA3N1fXXnutjhw5MujjkUhEkYj/B9gAACPbkH8O6NSpU2psbFRxcfFQbwoAMIIkPYAefvhh1dXV6ZNPPtHvf/973XbbbQqHw7rrrruSvSkAwAiW9F/Bffrpp7rrrrt08uRJTZw4UTfeeKP27NmjiRMnJntTAIARLOkB9Oqrryb7WyJF5UT6vGsyFfOuSQv512SFgg2s7Hf+/yWmRk5411yd1eJd09x/lXdNd4Dhr5KUlea//3pjGd41HbEs75rMsf7HHVITs+AAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYGPI/SIfRKxRy3jVdzn9gZfuZMd41Su/2r1GwIaY54dPeNT/++J+9a743tda75uPuIu8aScoNsP+6BjIDbctXJBJs0CxSD2dAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATTMNGYD1n/A+fgQCveZp7o941QU3MavWuydCAd030O0e8a2Y0Hfeu2dN5tXeNJLUFmEDe2R/xrgkyfTwW43XzaMEzCQAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwATDSBHYqR7/4ZNjQ33eNTHn/zrpk9MTvGsk6Z/GfOJdExum13GfDeR415RlfxZoW3v/VuZdc6Lbv7/MkP8g156eDO8apCbOgAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJhgGCkC6+7yH0aaEYoNQSfnG3ChQHUF4U7vmv9z8n8F2FKvd8XuzhneNf8S/dC7RpL+rXmWd03PGf8fJzlpPd41A6f5sTVacAYEADBBAAEATHgH0O7du3XLLbeopKREoVBI27dvT3jcOacnnnhCxcXFys7OVkVFhQ4fPpysfgEAo4R3AHV1dWnOnDnauHHjoI9v2LBBzz//vF588UXt3btXY8eO1ZIlS9TT4/+7XgDA6OX9bl5lZaUqKysHfcw5p+eee06PPfaYbr31VknSSy+9pMLCQm3fvl133nnn5XULABg1kvoeUFNTk1paWlRRURG/LxqNqry8XPX19YPW9Pb2qqOjI2EBAIx+SQ2glpYWSVJhYWHC/YWFhfHHzlVTU6NoNBpfSktLk9kSACBFmV8FV11drfb29vhy7Ngx65YAAMMgqQFUVFQkSWptbU24v7W1Nf7YuSKRiMaPH5+wAABGv6QGUFlZmYqKirRr1674fR0dHdq7d6/mz5+fzE0BAEY476vgTp06pSNHjsRvNzU16cCBA8rLy9PkyZO1du1a/eQnP9E111yjsrIyPf744yopKdGyZcuS2TcAYITzDqB9+/bp5ptvjt9et26dJGnFihXasmWLHnnkEXV1dWn16tVqa2vTjTfeqJ07dyorKyt5XQMARjzvAFq4cKGccxd8PBQK6emnn9bTTz99WY0h9cW6/IdCZmh4hpEGNTOz37tm58df9a6Zpo+8a+pPlHnX3J/3gXeNJPXH/H87n5V+xr8mNOBdE+oOe9cgNZlfBQcAuDIRQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEz4jzMG/kdal/9U4rFpwzMN+0ws2MTkcWn+fzYkoyE70LZ8HWvO864pvC4z0La6e/3r8sZ2e9eMCTANO3ya182jBc8kAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwwjRWDhnpB3TVbIv+b0QIZ3TX7klHdNUNH/HJ4Bq9mNEe+a8GL//R1UWsh512QEaC90xr8GqYkzIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYYRorAwr3+kyT7nP/Aykia//TJIIMxgxrf1DMs2xnbPHz/pjGRPu+asen+NWHvimBDcJGaOAMCAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABggmGkCCz99PBsJyb/4ZPRgM31uwHvmvSP/8u7xn8r0vhP/Id9BjUl52/eNX0x/9GiGSH/5zY8fLsBQ4wzIACACQIIAGDCO4B2796tW265RSUlJQqFQtq+fXvC4ytXrlQoFEpYli5dmqx+AQCjhHcAdXV1ac6cOdq4ceMF11m6dKmOHz8eX1555ZXLahIAMPp4X4RQWVmpysrKi64TiURUVFQUuCkAwOg3JO8B1dbWqqCgQNOnT9cDDzygkydPXnDd3t5edXR0JCwAgNEv6QG0dOlSvfTSS9q1a5d+9rOfqa6uTpWVlRoYGPzC05qaGkWj0fhSWlqa7JYAACko6Z8DuvPOO+Nfz5o1S7Nnz9a0adNUW1urRYsWnbd+dXW11q1bF7/d0dFBCAHAFWDIL8OeOnWq8vPzdeTIkUEfj0QiGj9+fMICABj9hjyAPv30U508eVLFxcVDvSkAwAji/Su4U6dOJZzNNDU16cCBA8rLy1NeXp6eeuopLV++XEVFRWpsbNQjjzyiq6++WkuWLElq4wCAkc07gPbt26ebb745fvuL929WrFihTZs26eDBg/r1r3+ttrY2lZSUaPHixfrxj3+sSCSSvK4BACOedwAtXLhQzrkLPv7b3/72shrCyBEOMO8zJ81/YGXvgP+1MuPCPd41QQ189tmwbCfrkwt/nCHZ8jK7vWv+2jdmCDo5Xyg2LJvBMGAWHADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADARNL/JDeuHOmnLzwV/UIy5D8NO4iw/HtLdQPHmodtWxMyT3nXBJmGnRHyfw0cOuNdghTFGRAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATDCNFYJmd/gM/w6GQd81Xc45716SFYt41kpQRGp5hqUG4/r5h29ZV6V3eNdPHtXrXZIX8fwRldI2+QbNXKs6AAAAmCCAAgAkCCABgggACAJgggAAAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmGAYKQJL7/Uf+JkW4DVPNHzauyaS1u9dI0n9biBQXarqjgXbDzlpPd414fThGRLKMNLRgzMgAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgggAAAJhhGipSXETrjXTM2rTfQtmLyH7CayvoVbHBnkP3X78KBtuUro5thpKMFZ0AAABMEEADAhFcA1dTU6Prrr1dOTo4KCgq0bNkyNTQ0JKzT09OjqqoqTZgwQePGjdPy5cvV2tqa1KYBACOfVwDV1dWpqqpKe/bs0TvvvKP+/n4tXrxYXV1d8XUeeughvfXWW3rjjTdUV1en5uZm3X777UlvHAAwsnldhLBz586E21u2bFFBQYH279+vBQsWqL29Xb/85S+1detWffvb35Ykbd68WV/5yle0Z88effOb30xe5wCAEe2y3gNqb2+XJOXl5UmS9u/fr/7+flVUVMTXmTFjhiZPnqz6+vpBv0dvb686OjoSFgDA6Bc4gGKxmNauXasbbrhBM2fOlCS1tLQoMzNTubm5CesWFhaqpaVl0O9TU1OjaDQaX0pLS4O2BAAYQQIHUFVVlQ4dOqRXX331shqorq5We3t7fDl27NhlfT8AwMgQ6IOoa9as0dtvv63du3dr0qRJ8fuLiorU19entra2hLOg1tZWFRUVDfq9IpGIIpFIkDYAACOY1xmQc05r1qzRtm3b9N5776msrCzh8blz5yojI0O7du2K39fQ0KCjR49q/vz5yekYADAqeJ0BVVVVaevWrdqxY4dycnLi7+tEo1FlZ2crGo3qvvvu07p165SXl6fx48frwQcf1Pz587kCDgCQwCuANm3aJElauHBhwv2bN2/WypUrJUk///nPlZaWpuXLl6u3t1dLlizRL37xi6Q0CwAYPbwCyLlLDwHMysrSxo0btXHjxsBNYWRI7x6ewZ0DAa6VCTLAVJJ6XLC6VNUZCza4My3k/9xmhAb8txPguc1sH13P0ZWMWXAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABOB/iIqIEnh7tSdShxWsCnQfx3wn+icyj4byA5UF3T/DYfMk6e9a4Znbjt8cQYEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABMNIkfJizv91UkYo2KDUf+8rCFSXqloGooHqstL6vGvCsaxA2/KV9lmbdw3DSFMTZ0AAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMAEAQQAMEEAAQBMMIwUgYUGnHdNfW94CDpJnv8cZcNIG3qKA9X9U/Yn3jVpAUZ+/t/uHO8a19XlXYPUxBkQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwwjRWCxLP/BohPSTnvXFGa0eddMTv+bd40ktZzJDVSXqv7jVLBhpJU5/+5d0+0i3jW54W7vGqXzY2u04AwIAGCCAAIAmPAKoJqaGl1//fXKyclRQUGBli1bpoaGhoR1Fi5cqFAolLDcf//9SW0aADDyeQVQXV2dqqqqtGfPHr3zzjvq7+/X4sWL1XXOH4hatWqVjh8/Hl82bNiQ1KYBACOf17t5O3fuTLi9ZcsWFRQUaP/+/VqwYEH8/jFjxqioqCg5HQIARqXLeg+ovb1dkpSXl5dw/8svv6z8/HzNnDlT1dXV6u6+8JUuvb296ujoSFgAAKNf4OsZY7GY1q5dqxtuuEEzZ86M33/33XdrypQpKikp0cGDB/Xoo4+qoaFBb7755qDfp6amRk899VTQNgAAI1TgAKqqqtKhQ4f0wQcfJNy/evXq+NezZs1ScXGxFi1apMbGRk2bNu2871NdXa1169bFb3d0dKi0tDRoWwCAESJQAK1Zs0Zvv/22du/erUmTJl103fLycknSkSNHBg2gSCSiSMT/A2wAgJHNK4Ccc3rwwQe1bds21dbWqqys7JI1Bw4ckCQVFwf7RDYAYHTyCqCqqipt3bpVO3bsUE5OjlpaWiRJ0WhU2dnZamxs1NatW/Wd73xHEyZM0MGDB/XQQw9pwYIFmj179pD8AwAAI5NXAG3atEnS2Q+b/v82b96slStXKjMzU++++66ee+45dXV1qbS0VMuXL9djjz2WtIYBAKOD96/gLqa0tFR1dXWX1RAA4MrAWFkEFopd/AXJYK7LzPau+dXJ8y9euZTmrKu8ayTp5aZ53jV5+jjQtoZDdrg/UN2vTt7oXZMRGvCu+eHEDy690rku8UIYIwfDSAEAJgggAIAJAggAYIIAAgCYIIAAACYIIACACQIIAGCCAAIAmCCAAAAmCCAAgAkCCABgggACAJgIuUuNuB5mHR0dikajWqhblR7KsG4HAODpjOtXrXaovb1d48ePv+B6nAEBAEwQQAAAEwQQAMAEAQQAMEEAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwES6dQPn+mI03Rn1Syk1pQ4A8I84o35Jf/95fiEpF0CdnZ2SpA/0G+NOAACXo7OzU9Fo9IKPp9w07FgspubmZuXk5CgUCiU81tHRodLSUh07duyiE1ZHO/bDWeyHs9gPZ7EfzkqF/eCcU2dnp0pKSpSWduF3elLuDCgtLU2TJk266Drjx4+/og+wL7AfzmI/nMV+OIv9cJb1frjYmc8XuAgBAGCCAAIAmBhRARSJRLR+/XpFIhHrVkyxH85iP5zFfjiL/XDWSNoPKXcRAgDgyjCizoAAAKMHAQQAMEEAAQBMEEAAABMjJoA2btyoL3/5y8rKylJ5ebn+8Ic/WLc07J588kmFQqGEZcaMGdZtDbndu3frlltuUUlJiUKhkLZv357wuHNOTzzxhIqLi5Wdna2KigodPnzYptkhdKn9sHLlyvOOj6VLl9o0O0Rqamp0/fXXKycnRwUFBVq2bJkaGhoS1unp6VFVVZUmTJigcePGafny5WptbTXqeGj8I/th4cKF5x0P999/v1HHgxsRAfTaa69p3bp1Wr9+vT788EPNmTNHS5Ys0YkTJ6xbG3bXXXedjh8/Hl8++OAD65aGXFdXl+bMmaONGzcO+viGDRv0/PPP68UXX9TevXs1duxYLVmyRD09PcPc6dC61H6QpKVLlyYcH6+88sowdjj06urqVFVVpT179uidd95Rf3+/Fi9erK6urvg6Dz30kN566y298cYbqqurU3Nzs26//XbDrpPvH9kPkrRq1aqE42HDhg1GHV+AGwHmzZvnqqqq4rcHBgZcSUmJq6mpMexq+K1fv97NmTPHug1Tkty2bdvit2OxmCsqKnLPPPNM/L62tjYXiUTcK6+8YtDh8Dh3Pzjn3IoVK9ytt95q0o+VEydOOEmurq7OOXf2uc/IyHBvvPFGfJ0//elPTpKrr6+3anPInbsfnHPuW9/6lvv+979v19Q/IOXPgPr6+rR//35VVFTE70tLS1NFRYXq6+sNO7Nx+PBhlZSUaOrUqbrnnnt09OhR65ZMNTU1qaWlJeH4iEajKi8vvyKPj9raWhUUFGj69Ol64IEHdPLkSeuWhlR7e7skKS8vT5K0f/9+9ff3JxwPM2bM0OTJk0f18XDufvjCyy+/rPz8fM2cOVPV1dXq7u62aO+CUm4Y6bk+//xzDQwMqLCwMOH+wsJC/fnPfzbqykZ5ebm2bNmi6dOn6/jx43rqqad000036dChQ8rJybFuz0RLS4skDXp8fPHYlWLp0qW6/fbbVVZWpsbGRv3oRz9SZWWl6uvrFQ6HrdtLulgsprVr1+qGG27QzJkzJZ09HjIzM5Wbm5uw7mg+HgbbD5J09913a8qUKSopKdHBgwf16KOPqqGhQW+++aZht4lSPoDwd5WVlfGvZ8+erfLyck2ZMkWvv/667rvvPsPOkAruvPPO+NezZs3S7NmzNW3aNNXW1mrRokWGnQ2NqqoqHTp06Ip4H/RiLrQfVq9eHf961qxZKi4u1qJFi9TY2Khp06YNd5uDSvlfweXn5yscDp93FUtra6uKioqMukoNubm5uvbaa3XkyBHrVsx8cQxwfJxv6tSpys/PH5XHx5o1a/T222/r/fffT/jzLUVFRerr61NbW1vC+qP1eLjQfhhMeXm5JKXU8ZDyAZSZmam5c+dq165d8ftisZh27dql+fPnG3Zm79SpU2psbFRxcbF1K2bKyspUVFSUcHx0dHRo7969V/zx8emnn+rkyZOj6vhwzmnNmjXatm2b3nvvPZWVlSU8PnfuXGVkZCQcDw0NDTp69OioOh4utR8Gc+DAAUlKrePB+iqIf8Srr77qIpGI27Jli/vjH//oVq9e7XJzc11LS4t1a8PqBz/4gautrXVNTU3ud7/7nauoqHD5+fnuxIkT1q0Nqc7OTvfRRx+5jz76yElyzz77rPvoo4/cX/7yF+eccz/96U9dbm6u27Fjhzt48KC79dZbXVlZmTt9+rRx58l1sf3Q2dnpHn74YVdfX++amprcu+++677+9a+7a665xvX09Fi3njQPPPCAi0ajrra21h0/fjy+dHd3x9e5//773eTJk917773n9u3b5+bPn+/mz59v2HXyXWo/HDlyxD399NNu3759rqmpye3YscNNnTrVLViwwLjzRCMigJxz7oUXXnCTJ092mZmZbt68eW7Pnj3WLQ27O+64wxUXF7vMzEz3pS99yd1xxx3uyJEj1m0Nuffff99JOm9ZsWKFc+7spdiPP/64KywsdJFIxC1atMg1NDTYNj0ELrYfuru73eLFi93EiRNdRkaGmzJlilu1atWoe5E22L9fktu8eXN8ndOnT7vvfe977qqrrnJjxoxxt912mzt+/Lhd00PgUvvh6NGjbsGCBS4vL89FIhF39dVXux/+8Ieuvb3dtvFz8OcYAAAmUv49IADA6EQAAQBMEEAAABMEEADABAEEADBBAAEATBBAAAATBBAAwAQBBAAwQQABAEwQQAAAEwQQAMDEfwNH5VgkZZXaeQAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.figure()\n", + "plt.imshow(img)\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 56, + "id": "8e1c07cc-b2bc-4902-a9a6-4ac7f02c5fe4", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "[ 2.5953586 3.9101725 0.65233815 3.2538052 1.270339 -3.0440047\n", + " 1.1500907 -3.7935097 -1.8807431 -3.321768 ]\n", + "predicted label: Trouser\n" + ] + } + ], + "source": [ + "print(predictions)\n", + "print(\"predicted label:\", classes[np.argmax(predictions)])" + ] + }, + { + "cell_type": "markdown", + "id": "a937adc9-508d-4ccd-b92d-8ecaa27ee4e4", + "metadata": {}, + "source": [ + "### Using Triton Inference Server\n", + "\n", + "Note: you can restart the kernel and run from this point to simulate running in a different node or environment." + ] + }, + { + "cell_type": "code", + "execution_count": 57, + "id": "53ca290a-ccc3-4923-a292-944921bab36d", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "from functools import partial\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import struct, col, array\n", + "from pyspark.sql.types import ArrayType, FloatType, Union, Dict" + ] + }, + { + "cell_type": "code", + "execution_count": 58, + "id": "8fa92fe4-2e04-4d82-a357-bfdfca38bd8c", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "# copy model to expected layout for Triton\n", + "rm -rf models\n", + "mkdir -p models/fashion_mnist/1\n", + "cp ts_model.pt models/fashion_mnist/1/model.pt\n", + "\n", + "# add config.pbtxt\n", + "cp models_config/fashion_mnist/config.pbtxt models/fashion_mnist/config.pbtxt" + ] + }, + { + "cell_type": "markdown", + "id": "d42b329c-5921-436f-bfca-a382a6762da4", + "metadata": {}, + "source": [ + "#### Start Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "5e869730-3597-4074-bab0-f87768f8996a", + "metadata": {}, + "outputs": [], + "source": [ + "num_executors = 1\n", + "triton_models_dir = \"{}/models\".format(os.getcwd())\n", + "nodeRDD = sc.parallelize(list(range(num_executors)), num_executors)\n", + "\n", + "def start_triton(it):\n", + " import docker\n", + " import time\n", + " import tritonclient.grpc as grpcclient\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " if containers:\n", + " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", + " else:\n", + " container=client.containers.run(\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", + " detach=True,\n", + " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", + " name=\"spark-triton\",\n", + " network_mode=\"host\",\n", + " remove=True,\n", + " shm_size=\"64M\",\n", + " volumes={triton_models_dir: {\"bind\": \"/models\", \"mode\": \"ro\"}}\n", + " )\n", + " print(\">>>> starting triton: {}\".format(container.short_id))\n", + "\n", + " # wait for triton to be running\n", + " time.sleep(15)\n", + " client = grpcclient.InferenceServerClient(\"localhost:8001\")\n", + " ready = False\n", + " while not ready:\n", + " try:\n", + " ready = client.is_server_ready()\n", + " except Exception as e:\n", + " time.sleep(5)\n", + " \n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(start_triton).collect()" + ] + }, + { + "cell_type": "markdown", + "id": "30a4362d-7514-4b84-b238-f704a97e1e72", + "metadata": {}, + "source": [ + "#### Run inference" + ] + }, + { + "cell_type": "code", + "execution_count": 60, + "id": "ab94d4d1-dac6-4474-9eb0-59478aa98f7d", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "1" + ] + }, + "execution_count": 60, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = spark.read.parquet(\"fashion_mnist_1\")\n", + "len(df.columns)" + ] + }, + { + "cell_type": "code", + "execution_count": 61, + "id": "12b5f2fc-52e9-428a-b683-6ab1b639aa24", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "data": { + "text/plain": [ + "StructType([StructField('data', ArrayType(FloatType(), True), True)])" + ] + }, + "execution_count": 61, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df.schema" + ] + }, + { + "cell_type": "code", + "execution_count": 62, + "id": "960657d0-31c9-4df6-8eb8-ac3d23137f7a", + "metadata": {}, + "outputs": [], + "source": [ + "def triton_fn(triton_uri, model_name):\n", + " import numpy as np\n", + " import tritonclient.grpc as grpcclient\n", + " \n", + " np_types = {\n", + " \"BOOL\": np.dtype(np.bool_),\n", + " \"INT8\": np.dtype(np.int8),\n", + " \"INT16\": np.dtype(np.int16),\n", + " \"INT32\": np.dtype(np.int32),\n", + " \"INT64\": np.dtype(np.int64),\n", + " \"FP16\": np.dtype(np.float16),\n", + " \"FP32\": np.dtype(np.float32),\n", + " \"FP64\": np.dtype(np.float64),\n", + " \"FP64\": np.dtype(np.double),\n", + " \"BYTES\": np.dtype(object)\n", + " }\n", + "\n", + " client = grpcclient.InferenceServerClient(triton_uri)\n", + " model_meta = client.get_model_metadata(model_name)\n", + " \n", + " def predict(inputs):\n", + " if isinstance(inputs, np.ndarray):\n", + " # single ndarray input\n", + " request = [grpcclient.InferInput(model_meta.inputs[0].name, inputs.shape, model_meta.inputs[0].datatype)]\n", + " request[0].set_data_from_numpy(inputs.astype(np_types[model_meta.inputs[0].datatype]))\n", + " else:\n", + " # dict of multiple ndarray inputs\n", + " request = [grpcclient.InferInput(i.name, inputs[i.name].shape, i.datatype) for i in model_meta.inputs]\n", + " for i in request:\n", + " i.set_data_from_numpy(inputs[i.name()].astype(np_types[i.datatype()]))\n", + " \n", + " response = client.infer(model_name, inputs=request)\n", + " \n", + " if len(model_meta.outputs) > 1:\n", + " # return dictionary of numpy arrays\n", + " return {o.name: response.as_numpy(o.name) for o in model_meta.outputs}\n", + " else:\n", + " # return single numpy array\n", + " return response.as_numpy(model_meta.outputs[0].name)\n", + " \n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 63, + "id": "0262fd4a-9845-44b9-8c75-1c105e7deeca", + "metadata": {}, + "outputs": [], + "source": [ + "mnist = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"fashion_mnist\"),\n", + " input_tensor_shapes=[[784]],\n", + " return_type=ArrayType(FloatType()),\n", + " batch_size=1024)" + ] + }, + { + "cell_type": "code", + "execution_count": 64, + "id": "fc5f6baa-052e-4b89-94b6-4821cf01952a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 326 ms, sys: 39.6 ms, total: 365 ms\n", + "Wall time: 1.77 s\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(struct(df.columns))).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 65, + "id": "a85dea35-e41d-482d-8a8f-52d3c108f038", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 199 ms, sys: 63.2 ms, total: 262 ms\n", + "Wall time: 1.21 s\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(*df.columns)).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 66, + "id": "bc3f0dbe-c52b-41d6-8097-8cebaa5ee5a8", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 195 ms, sys: 25.4 ms, total: 220 ms\n", + "Wall time: 1.21 s\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(*[col(c) for c in df.columns])).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 67, + "id": "99fb5e8d", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Predicted label: Bag\n" + ] + }, + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGdCAYAAABU0qcqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAknklEQVR4nO3de3SV9Z3v8c9OSDa3ZMcQcpNAAyi0cumUSsqoFEsOkM64QDkdb3MOuDow0uCqUqsnPVZq27PS4hrrqUNxrbNaqKvihXNERsdiBSWMCnRAGMZeUsAoYSChYJMNCUl2sn/nD8bMREH4/kzyS8L7tdZei+z9fHh+efIknzzZO99EnHNOAAD0spTQCwAAXJooIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBDAq9gA9LJpM6evSoMjIyFIlEQi8HAGDknNOpU6dUWFiolJTzX+f0uQI6evSoioqKQi8DAPAJ1dbWatSoUed9vM8VUEZGhiTpWn1Zg5QWeDXodr11VcuEKSCYdiX0ul7q/Hp+Pj1WQKtXr9bDDz+suro6TZ06VY899pimT59+wdwHP3YbpDQNilBAA06v/ViVAgKC+fdPvws9jdIjL0J45plntGLFCq1cuVJvvfWWpk6dqrlz5+r48eM9sTsAQD/UIwX0yCOPaMmSJbrjjjv0mc98Ro8//riGDh2qn/3sZz2xOwBAP9TtBdTW1qY9e/aotLT0P3aSkqLS0lLt2LHjI9u3trYqHo93uQEABr5uL6ATJ06oo6NDeXl5Xe7Py8tTXV3dR7avrKxULBbrvPEKOAC4NAT/RdSKigo1NjZ23mpra0MvCQDQC7r9VXA5OTlKTU1VfX19l/vr6+uVn5//ke2j0aii0Wh3LwMA0Md1+xVQenq6pk2bpq1bt3bel0wmtXXrVs2YMaO7dwcA6Kd65PeAVqxYoUWLFunzn/+8pk+frkcffVRNTU264447emJ3AIB+qEcK6Oabb9Yf//hHPfjgg6qrq9NnP/tZbd68+SMvTAAAXLoizvWtmSXxeFyxWEyzNJ9JCBiwUscXmzPH/s7+XGnu/N+bM8An1e4S2qZNamxsVGZm5nm3C/4qOADApYkCAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQfTINGygv6r5gf1vVt0/f6M5U9Vw/gGN5zM+7Yw5k7/fnpGkX2yYbc4Ufe9Nr33h0sUVEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIJgGnYfFhlk//C4jg77jpyzZzxF0tLNGZdoM2cGFY8xZyRpy20PmzNf/NXd5syVf7PbnKk3J6TdN1/vkZIe/O5T5sza7/kdc7NIxJ7pxXMcF48rIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAIIuJc35rSF4/HFYvFNEvzNSiSFno53cdjgGIkNdWc8RpG6qtvnTpd/GHtNK/c2KI/mjODSg977asvO/NysTlzw+X7zZktkzLMGfR97S6hbdqkxsZGZWZmnnc7roAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIIhBoRdwyfAY3Ona2+378Rh62peHikpSytRPmzNbvvS/vfZV+ssV5syV8hhGmmIfNBtJsX9svc4hSWk/zDZnbl73L+bMhsXfNGcuW7fDnEHfxBUQACAICggAEES3F9B3vvMdRSKRLreJEyd2924AAP1cjzwHdNVVV2nLli3/sZNBPNUEAOiqR5ph0KBBys/P74n/GgAwQPTIc0AHDhxQYWGhxo4dq9tvv12HD5//VUKtra2Kx+NdbgCAga/bC6ikpETr1q3T5s2btWbNGtXU1Oi6667TqVOnzrl9ZWWlYrFY562oqKi7lwQA6IO6vYDKysr0la98RVOmTNHcuXP10ksvqaGhQc8+++w5t6+oqFBjY2Pnrba2truXBADog3r81QFZWVm68sordfDgwXM+Ho1GFY1Ge3oZAIA+psd/D+j06dM6dOiQCgoKenpXAIB+pNsL6N5771VVVZXeffddvfnmm7rxxhuVmpqqW2+9tbt3BQDox7r9R3BHjhzRrbfeqpMnT2rkyJG69tprtXPnTo0cObK7dwUA6Me6vYCefvrp7v4vYeEzWNRjMKYkKdlhjsRv/YI5M2b5H8yZx09eZ85IUuGrvTSdyiXNkUh0qH03nsNI675gf172nUSmOfPMQw+bMyv/9svmTP0Mfr2jL2IWHAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAE0eN/kK7XRCL2jM/gzt7kMyTUY8ilz1BRX1d/Y485U92YZ840t6ebM5I0/NmdXjmrSKrnANheknbKnnmz6Qpz5tE/fcqcuWvUFnPmgduWmDOSlLne43zora9FPvvx3VcP4QoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQQycadh9ncfkWp+JyS7Re5Oth20fac60O/uY5dQU+4TvdzeNNWckqUB1Xjkr1+HxcWpLdP9CziPvsTfNmW9VVJsz1x69ypxZeWC+OXPz/9xszkjSyy8UmTPJUx6jxH34TrXuQ385gCsgAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAhi4Awj9RiWF0lL99tVos0jZF+f1348HL3vz71y38x91px58t++YM4kZR+eWPCIfZhmr+rD54OvXzWnmTP/fcxOc2bV3jnmTGqR3zDNjF/av0Y0Xuu1q94T8bjucD0z5JgrIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAIIuKcx1TEHhSPxxWLxTRL8zUoYh9uOJAcX24fEtqaZd/Pr5ausockrWv4vDkzKv19c+Z7v7zJnIn9wT7AVJKu/5td5szG33zWnBn0b1FzJqXN432K+H16++yrdUTSnBk54YQ5E4u2mDNn2v2+lqwc/w/mzLINS82Z4v+xw5zpy9pdQtu0SY2NjcrMzDzvdlwBAQCCoIAAAEGYC2j79u264YYbVFhYqEgkoueff77L4845PfjggyooKNCQIUNUWlqqAwcOdNd6AQADhLmAmpqaNHXqVK1evfqcj69atUo//vGP9fjjj2vXrl0aNmyY5s6dq5YW+89tAQADl/kvopaVlamsrOycjznn9Oijj+qBBx7Q/PnzJUlPPPGE8vLy9Pzzz+uWW275ZKsFAAwY3focUE1Njerq6lRaWtp5XywWU0lJiXbsOPerPFpbWxWPx7vcAAADX7cWUF1dnSQpLy+vy/15eXmdj31YZWWlYrFY562oqKg7lwQA6KOCvwquoqJCjY2Nnbfa2trQSwIA9IJuLaD8/HxJUn19fZf76+vrOx/7sGg0qszMzC43AMDA160FVFxcrPz8fG3durXzvng8rl27dmnGjBnduSsAQD9nfhXc6dOndfDgwc63a2pqtG/fPmVnZ2v06NG6++679f3vf19XXHGFiouL9e1vf1uFhYVasGBBd64bANDPmQto9+7duv766zvfXrFihSRp0aJFWrdune677z41NTVp6dKlamho0LXXXqvNmzdr8ODB3bdqAEC/d0kPI31nld+PBf/uxp+bMyv++a/MmfT0dnPmB1OfM2c2N0wxZyQpRfZT53eNeRfe6EPe23u5OdNxWcKckaTBtenmTOY79uOQ0m7PdKTbB4Qmzd9inuVS7Zlkmn19kaT9OJye2WzOfLboiDkjSUdPx8yZa/LeMWfeet/+6t8j72eZM5I0+iv/6pWzYBgpAKBPo4AAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAjPWbkDw29v/3uv3MKDf2HO5Pyj/c9RnF54ypz52dHrzJnGNr8/lXFH0RvmzIm2YeZMzeCkOaMO+2RmSWq7zL6vxFf+ZM6MijWaMyOjp82ZaKp9orokZQ2yT5xOeIzQbuqImjMzM6vt+0na9yNJbw4ab86kyn4ODRvUZs68NH2NOSNJt95+rzkTe3Kn174uhCsgAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAhiwAwjbblhujmTFtnnt69v5ZszOf/rPXPm5fHPmTMPn7Afh6Ep9kGIkrTy9QXmTErcfsq5LI+Bmh7zS8/uK2HOxA9cZs4cbBhhztTaZ54qtdXZQ546ovYBsM5jZuzr6Z8zZ25f/Ip9R5Kuy/qDOfO5wYfNmZfTrjJn/uKf7zRnJOlvKn5lzrz8ZKbXvi6EKyAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACGLADCOt/3zvvStZP6w1Z/4y51/MmZ822AcU/lXWP5sz3639S3NGki7/Zao5kxjqMX1SaeZEJOk3hNOl2NfXkW7fTzLNvj6ftbVm+RxvSR6xiMfMWJ/9DP83+6TZx//pevuOJP1h/hpz5jdt9ndqUWy/OfOPmZPNGUn669i/mjO/+rOlpu0jHa3Sv2y64HZcAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEANmGGna6d7b172XbzZnNvxpujmTmx43Z7556L+aM81/f7k5I0lncu3fv/gMx0xtM0eUTPUbwpnSSwM1fQZ3OvvsV6+1SVL7EL+clc9xODXaft4VVNn3I0lfmzbTnCkc3GDOVJ/OM2cWXr7XnJGkESn2D+6Zy4eZtm9PpEoXMX+ZKyAAQBAUEAAgCHMBbd++XTfccIMKCwsViUT0/PPPd3l88eLFikQiXW7z5s3rrvUCAAYIcwE1NTVp6tSpWr169Xm3mTdvno4dO9Z5e+qppz7RIgEAA4/5RQhlZWUqKyv72G2i0ajy8/O9FwUAGPh65Dmgbdu2KTc3VxMmTNCyZct08uTJ827b2tqqeDze5QYAGPi6vYDmzZunJ554Qlu3btUPf/hDVVVVqaysTB0dHefcvrKyUrFYrPNWVFTU3UsCAPRB3f57QLfcckvnvydPnqwpU6Zo3Lhx2rZtm2bPnv2R7SsqKrRixYrOt+PxOCUEAJeAHn8Z9tixY5WTk6ODBw+e8/FoNKrMzMwuNwDAwNfjBXTkyBGdPHlSBQUFPb0rAEA/Yv4R3OnTp7tczdTU1Gjfvn3Kzs5Wdna2HnroIS1cuFD5+fk6dOiQ7rvvPo0fP15z587t1oUDAPo3cwHt3r1b119/fefbHzx/s2jRIq1Zs0b79+/Xz3/+czU0NKiwsFBz5szR9773PUWj0e5bNQCg3zMX0KxZs+ScO+/jL7/88idakK9Bzb23rwNt9t9xejjfPjjw/522Px/W+LP/Ys4kY34TKxPD7LnIuV8M+bE60u0Z3yGcH3Nqn39XvTRY1GtAqOdxGHTGnknxGBrrcz74HLuOqN+B+PX6qebMhhUPmzP/lD7OnLl6yLvmjCTFk0lzZlj1CdP27R2tF7Uds+AAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQRLf/Se5QhpywT3j1NS7tuDlzpN0+Xvi+l+40ZzJG2L+nSAwzRyRJ6XF7xnl8y9Mx2J6Rx1RrSWof6rErj+nMPuvzmbrtKzHcvsCkx1eT1Db7lOqUixu03EX8U37TsDMO24/Dtw7PN2f+77gt5sy2Mx4nq6TC1FPmTMeBd2zbu8RFbccVEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEMWCGkWYeOm3OJFyH177GprWYMzN3LDNn8t+wD0L800RzxHvIZUuOPdMRtb9P0fc9Bkl6fmvlMyy1faj9fWrPsJ97KcMvbsDjf5Zs9ZmUKkWPppkzaaftHyevY+cxnDa11W8Y6Zlce+6d9VeYM7+7/x/MGWm4R0a6LGWIOZM6Ybxpe9fRKh248HZcAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEANmGGlKc5s5kxbxG9T4r22Z5szwX9kHB/5poscAxaQ90jHEPhBSktqy7DuLnrAf8/RT9vWlLzhuzkjSycZh5kxHm/3TKP1w1JzJ3m7/fjHi96HVmWz7uddc6DFY1GMYqXzmivrNIlUiw76+ISn2j9P8nXeaM7+c8RNzRpLaZT/3IgnbxOJI8uK25woIABAEBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIIYMMNI1d5hjjQmz3jt6m93fs2cSRtln4bYmmN/n1Ja7fuJJPwmNabnN5szY38UN2fa3ztiztRHS8wZSRq/9aQ9dLTWnskdYY68d1OuOdM8xjZE8gORofbhvu6Mx3DfpMf52m7PdKT6TWV1g+y55lH2TMZO+7DiuulDzRlJGpdmv+5of+dd2/YucVHbcQUEAAiCAgIABGEqoMrKSl199dXKyMhQbm6uFixYoOrq6i7btLS0qLy8XCNGjNDw4cO1cOFC1dfXd+uiAQD9n6mAqqqqVF5erp07d+qVV15RIpHQnDlz1NTU1LnNPffcoxdeeEEbNmxQVVWVjh49qptuuqnbFw4A6N9ML0LYvHlzl7fXrVun3Nxc7dmzRzNnzlRjY6N++tOfav369frSl74kSVq7dq0+/elPa+fOnfrCF77QfSsHAPRrn+g5oMbGRklSdna2JGnPnj1KJBIqLS3t3GbixIkaPXq0duzYcc7/o7W1VfF4vMsNADDweRdQMpnU3XffrWuuuUaTJk2SJNXV1Sk9PV1ZWVldts3Ly1NdXd05/5/KykrFYrHOW1FRke+SAAD9iHcBlZeX6+2339bTTz/9iRZQUVGhxsbGzlttrcfvVAAA+h2vX0Rdvny5XnzxRW3fvl2jRo3qvD8/P19tbW1qaGjochVUX1+v/Pz8c/5f0WhU0WjUZxkAgH7MdAXknNPy5cu1ceNGvfrqqyouLu7y+LRp05SWlqatW7d23lddXa3Dhw9rxowZ3bNiAMCAYLoCKi8v1/r167Vp0yZlZGR0Pq8Ti8U0ZMgQxWIxffWrX9WKFSuUnZ2tzMxM3XXXXZoxYwavgAMAdGEqoDVr1kiSZs2a1eX+tWvXavHixZKkH/3oR0pJSdHChQvV2tqquXPn6ic/+Um3LBYAMHCYCsi5Cw/ZGzx4sFavXq3Vq1d7L8pLqv31FC82jbrwRufgkh4Zj3mfqc0eQwOHewwwjfi9FiXZan8KsfnKkeZMes175szlzx4yZySp/i/GmjMnr8kwZ7JGnDZnWk/bh+dG3k83ZyRJDWn2ffnMtPU59XwyfrNIFUnYd+aG2wfANhfa9/PXW/7WnJGkmr/8P+ZM6ohs0/Yu2Sa9f+HtmAUHAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAILz+Impf1PG7A+bMsJRWr309cc1PzZm/Tiw1ZyLNqeZMaixhziQ7/CYmuxb76RNf3mjOpNx1pTnT3Gqf5ixJzp2yh94fYo40vptlzkTsg86lNM8x0PZTz2vitEv1CHlMo4/4jKOX5AbbD3pqg/3zomOY/Z1Kr++9L98tf1Z84Y3+k/b2Fum1C2/HFRAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABDFghpH6yEpp9sqlekxd/NF1T5szhYP+ZM784uSfmzMvvjHNnJEkddgHPL5/JMucSW22f5/kevFbq4jHQE2XZh8+6fxmxnqJtHsM7/SZ9+kzK9VjPy7FcyirxzmejHruyygl4TdgtTnZZs60Zdmqoj1xcdtzBQQACIICAgAEQQEBAIKggAAAQVBAAIAgKCAAQBAUEAAgCAoIABAEBQQACIICAgAEQQEBAIKggAAAQVzSw0i3nLrKKzdl6GFz5nKPwaIT0trNmU8NPmnO/LdZ/2TOSNL631xtDx0eYo6k2A+DEjH7sE9JingMn4wk7ZmUM36DJK1c7+xGkhTxmMHpUuwL9Bk067O2s/vyead8ziH7bga12DOSVNPeYc4MPpEwbd/efnHbcwUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFc2sNIj07wysVGN5sz49L+aM78In6lOZMWsQ8aLMvYb85I0pTP15ozI0pOmzN5qfbMxvifmTOS9NJR+4Da9qT9+7jmtjRzJumxn2iabYjkB4Z4DML1OQ6DUuxTOH3mip5uiXqkpNgQ+8TPwuGN5kxbR6o5k/SZyirpjTPjzJljMwabtu9olXQRM465AgIABEEBAQCCMBVQZWWlrr76amVkZCg3N1cLFixQdXV1l21mzZqlSCTS5XbnnXd266IBAP2fqYCqqqpUXl6unTt36pVXXlEikdCcOXPU1NTUZbslS5bo2LFjnbdVq1Z166IBAP2f6UUImzdv7vL2unXrlJubqz179mjmzJmd9w8dOlT5+fnds0IAwID0iZ4Damw8+2qP7OzsLvc/+eSTysnJ0aRJk1RRUaHm5vO/aqy1tVXxeLzLDQAw8Hm/DDuZTOruu+/WNddco0mTJnXef9ttt2nMmDEqLCzU/v37df/996u6ulrPPffcOf+fyspKPfTQQ77LAAD0U94FVF5errfffluvv/56l/uXLl3a+e/JkyeroKBAs2fP1qFDhzRu3Edff15RUaEVK1Z0vh2Px1VUVOS7LABAP+FVQMuXL9eLL76o7du3a9SoUR+7bUlJiSTp4MGD5yygaDSqaNTvl8QAAP2XqYCcc7rrrru0ceNGbdu2TcXFxRfM7Nu3T5JUUFDgtUAAwMBkKqDy8nKtX79emzZtUkZGhurq6iRJsVhMQ4YM0aFDh7R+/Xp9+ctf1ogRI7R//37dc889mjlzpqZMmdIj7wAAoH8yFdCaNWsknf1l0/9s7dq1Wrx4sdLT07VlyxY9+uijampqUlFRkRYuXKgHHnig2xYMABgYzD+C+zhFRUWqqqr6RAsCAFwaLulp2EM8JwUvy/qNOXMwETFnyrPs06b92CfxnmX/na3mZJs5MzRlqDnz6ZzqC290Dssu22vOXJZqX5+PN1rsk6Mbkn5ry0+1f2wHe0xi75D986LF2c/XkSmt5owkFacNN2f2tNrP8fFp9mO3oyXLnJGk9X8sMWdGVb5p2r7dJXTgIrZjGCkAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQBAUEAAiCAgIABHFpDyP9ut9fYi2b8HVzJtpgH3yaTOud7w86on77OXK9PRfJsw+FzHhziDlT+Muj5owkuVT7+9Rx2TBzJvVUizmjY8fNEZdot+9HUmSofYhpZLjH4NMLTNg/p3b74E5fzVfZ/5Cmz+ft8D2HzZn2Y3XmzFn2QbM9hSsgAEAQFBAAIAgKCAAQBAUEAAiCAgIABEEBAQCCoIAAAEFQQACAICggAEAQFBAAIAgKCAAQRJ+bBef+fTZUuxKSx5go07467HPJJKk9YZ/jldruMQsu0kuz4FL89pNs8ZgF12w/5h1tEXOmPen3sXUe35N1tKfa9+Nz7rk2e8R5zoJL2r80RJL24+A1Cy7Ze7Pg2tvtn+tJj3OoPWn/2LY7+9eU3tKus2tzF/j4RtyFtuhlR44cUVFRUehlAAA+odraWo0aNeq8j/e5Akomkzp69KgyMjIUiXT9zjcej6uoqEi1tbXKzMwMtMLwOA5ncRzO4jicxXE4qy8cB+ecTp06pcLCQqV8zE9Y+tyP4FJSUj62MSUpMzPzkj7BPsBxOIvjcBbH4SyOw1mhj0MsFrvgNrwIAQAQBAUEAAiiXxVQNBrVypUrFY36/SXTgYLjcBbH4SyOw1kch7P603Hocy9CAABcGvrVFRAAYOCggAAAQVBAAIAgKCAAQBD9poBWr16tT33qUxo8eLBKSkr061//OvSSet13vvMdRSKRLreJEyeGXlaP2759u2644QYVFhYqEono+eef7/K4c04PPvigCgoKNGTIEJWWlurAgQNhFtuDLnQcFi9e/JHzY968eWEW20MqKyt19dVXKyMjQ7m5uVqwYIGqq6u7bNPS0qLy8nKNGDFCw4cP18KFC1VfXx9oxT3jYo7DrFmzPnI+3HnnnYFWfG79ooCeeeYZrVixQitXrtRbb72lqVOnau7cuTp+/HjopfW6q666SseOHeu8vf7666GX1OOampo0depUrV69+pyPr1q1Sj/+8Y/1+OOPa9euXRo2bJjmzp2rlhb7IMm+7ELHQZLmzZvX5fx46qmnenGFPa+qqkrl5eXauXOnXnnlFSUSCc2ZM0dNTU2d29xzzz164YUXtGHDBlVVVeno0aO66aabAq66+13McZCkJUuWdDkfVq1aFWjF5+H6genTp7vy8vLOtzs6OlxhYaGrrKwMuKret3LlSjd16tTQywhKktu4cWPn28lk0uXn57uHH364876GhgYXjUbdU089FWCFvePDx8E55xYtWuTmz58fZD2hHD9+3ElyVVVVzrmzH/u0tDS3YcOGzm1+97vfOUlux44doZbZ4z58HJxz7otf/KL7+te/Hm5RF6HPXwG1tbVpz549Ki0t7bwvJSVFpaWl2rFjR8CVhXHgwAEVFhZq7Nixuv3223X48OHQSwqqpqZGdXV1Xc6PWCymkpKSS/L82LZtm3JzczVhwgQtW7ZMJ0+eDL2kHtXY2ChJys7OliTt2bNHiUSiy/kwceJEjR49ekCfDx8+Dh948sknlZOTo0mTJqmiokLNzc0hlndefW4Y6YedOHFCHR0dysvL63J/Xl6efv/73wdaVRglJSVat26dJkyYoGPHjumhhx7Sddddp7ffflsZGRmhlxdEXV2dJJ3z/PjgsUvFvHnzdNNNN6m4uFiHDh3St771LZWVlWnHjh1KTfX4Wz19XDKZ1N13361rrrlGkyZNknT2fEhPT1dWVlaXbQfy+XCu4yBJt912m8aMGaPCwkLt379f999/v6qrq/Xcc88FXG1Xfb6A8B/Kyso6/z1lyhSVlJRozJgxevbZZ/XVr3414MrQF9xyyy2d/548ebKmTJmicePGadu2bZo9e3bAlfWM8vJyvf3225fE86Af53zHYenSpZ3/njx5sgoKCjR79mwdOnRI48aN6+1lnlOf/xFcTk6OUlNTP/Iqlvr6euXn5wdaVd+QlZWlK6+8UgcPHgy9lGA+OAc4Pz5q7NixysnJGZDnx/Lly/Xiiy/qtdde6/LnW/Lz89XW1qaGhoYu2w/U8+F8x+FcSkpKJKlPnQ99voDS09M1bdo0bd26tfO+ZDKprVu3asaMGQFXFt7p06d16NAhFRQUhF5KMMXFxcrPz+9yfsTjce3ateuSPz+OHDmikydPDqjzwzmn5cuXa+PGjXr11VdVXFzc5fFp06YpLS2ty/lQXV2tw4cPD6jz4ULH4Vz27dsnSX3rfAj9KoiL8fTTT7toNOrWrVvnfvvb37qlS5e6rKwsV1dXF3ppveob3/iG27Ztm6upqXFvvPGGKy0tdTk5Oe748eOhl9ajTp065fbu3ev27t3rJLlHHnnE7d2717333nvOOed+8IMfuKysLLdp0ya3f/9+N3/+fFdcXOzOnDkTeOXd6+OOw6lTp9y9997rduzY4WpqatyWLVvc5z73OXfFFVe4lpaW0EvvNsuWLXOxWMxt27bNHTt2rPPW3Nzcuc2dd97pRo8e7V599VW3e/duN2PGDDdjxoyAq+5+FzoOBw8edN/97nfd7t27XU1Njdu0aZMbO3asmzlzZuCVd9UvCsg55x577DE3evRol56e7qZPn+527twZekm97uabb3YFBQUuPT3dXX755e7mm292Bw8eDL2sHvfaa685SR+5LVq0yDl39qXY3/72t11eXp6LRqNu9uzZrrq6Ouyie8DHHYfm5mY3Z84cN3LkSJeWlubGjBnjlixZMuC+STvX+y/JrV27tnObM2fOuK997Wvusssuc0OHDnU33nijO3bsWLhF94ALHYfDhw+7mTNnuuzsbBeNRt348ePdN7/5TdfY2Bh24R/Cn2MAAATR558DAgAMTBQQACAICggAEAQFBAAIggICAARBAQEAgqCAAABBUEAAgCAoIABAEBQQACAICggAEAQFBAAI4v8DIE1CeiobCX8AAAAASUVORK5CYII=", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Sample prediction\n", + "sample = preds[0]\n", + "predictions = sample.preds\n", + "img = sample.data\n", + "\n", + "img = np.array(img).reshape(28,28)\n", + "plt.figure()\n", + "plt.imshow(img)\n", + "\n", + "print(\"Predicted label:\", classes[np.argmax(predictions)])" + ] + }, + { + "cell_type": "markdown", + "id": "7a26690a-9dc4-4c36-9904-568d73e2be3c", + "metadata": { + "tags": [] + }, + "source": [ + "#### Stop Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ab2fe42f-a072-4370-bac2-52fd95363530", + "metadata": {}, + "outputs": [], + "source": [ + "def stop_triton(it):\n", + " import docker\n", + " import time\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " print(\">>>> stopping containers: {}\".format([c.short_id for c in containers]))\n", + " if containers:\n", + " container=containers[0]\n", + " container.stop(timeout=120)\n", + "\n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(stop_triton).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 69, + "id": "a0608fff-7cfb-489e-96c9-8e1d92e57562", + "metadata": {}, + "outputs": [], + "source": [ + "spark.stop()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "08de2664-3d60-487b-90da-6d0f3b8b9203", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "spark-dl-torch", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/regression.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/regression_torch.ipynb similarity index 55% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/regression.ipynb rename to examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/regression_torch.ipynb index 3e537d42f..d2fd9157d 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/regression.ipynb +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/pytorch/regression_torch.ipynb @@ -2,12 +2,15 @@ "cells": [ { "cell_type": "markdown", - "id": "f04d288a-f4bf-4256-8ef8-fbc3ab92f24e", - "metadata": { - "tags": [] - }, + "id": "792d95f9", + "metadata": {}, "source": [ - "Based on: https://github.com/christianversloot/machine-learning-articles/blob/main/how-to-create-a-neural-network-for-regression-with-pytorch.md" + "# PySpark PyTorch Inference\n", + "\n", + "### Regression\n", + "Based on: https://github.com/christianversloot/machine-learning-articles/blob/main/how-to-create-a-neural-network-for-regression-with-pytorch.md \n", + "\n", + "For the first MLP (array inputs) we'll also demonstrate accelerated inference on GPU with Torch-TensorRT. " ] }, { @@ -29,13 +32,13 @@ { "cell_type": "code", "execution_count": 2, - "id": "cf02ba0a-8384-42b5-917c-53889b4a6471", + "id": "6d5bc0c7", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "" + "'2.4.1+cu121'" ] }, "execution_count": 2, @@ -44,12 +47,53 @@ } ], "source": [ - "torch.manual_seed(42)" + "torch.__version__" ] }, { "cell_type": "code", "execution_count": 3, + "id": "cf02ba0a-8384-42b5-917c-53889b4a6471", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 3, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "torch.manual_seed(42)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "bb5c10ab", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Using cuda device\n" + ] + } + ], + "source": [ + "# Get cpu or gpu device for training.\n", + "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", + "print(f\"Using {device} device\")" + ] + }, + { + "cell_type": "code", + "execution_count": 5, "id": "2bee64cf-a44a-4aff-82db-c64ee3a8b0e8", "metadata": {}, "outputs": [], @@ -59,7 +103,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 6, "id": "8644e508-5e4c-4cdd-9ed1-9235887d9659", "metadata": {}, "outputs": [], @@ -82,7 +126,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 7, "id": "cc6b55c3-dc7b-4831-9943-83efd48091bf", "metadata": {}, "outputs": [], @@ -94,7 +138,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 8, "id": "d868f39d-4695-4110-91d2-6f7a09d73b93", "metadata": {}, "outputs": [ @@ -115,7 +159,7 @@ " 2.0530])]" ] }, - "execution_count": 6, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } @@ -126,7 +170,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 9, "id": "9a441b60-dca4-44d2-bc1c-aa7336d704bb", "metadata": {}, "outputs": [], @@ -148,7 +192,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 10, "id": "15cff2b4-9d23-4d2b-808a-a5edb8eda135", "metadata": { "scrolled": true, @@ -157,7 +201,7 @@ "outputs": [], "source": [ "# Initialize the MLP\n", - "mlp = MLP()\n", + "mlp = MLP().to(device)\n", "\n", "# Define the loss function and optimizer\n", "loss_function = nn.L1Loss()\n", @@ -166,7 +210,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 11, "id": "5e2db3f9-5db8-4b42-89ad-e77f23c4c1fe", "metadata": { "scrolled": true, @@ -256,6 +300,7 @@ "\n", " # Get and prepare inputs\n", " inputs, targets = data\n", + " inputs, targets = inputs.to(device), targets.to(device)\n", " targets = targets.reshape((targets.shape[0], 1))\n", "\n", " # Zero the gradients\n", @@ -286,31 +331,59 @@ }, { "cell_type": "markdown", - "id": "ace480ba-9316-49a4-9763-dc0f61f66989", + "id": "352539f5", "metadata": {}, "source": [ - "### Save Model" + "### Save Model State Dict\n", + "This saves the serialized object to disk using pickle." ] }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 12, "id": "b950a3ed-ffe1-477f-a84f-f71c85dbf9ce", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saved PyTorch Model State to housing_model.pt\n" + ] + } + ], "source": [ - "torch.save(mlp, \"housing_model.pt\")" + "torch.save(mlp.state_dict(), \"housing_model.pt\")\n", + "print(\"Saved PyTorch Model State to housing_model.pt\")" + ] + }, + { + "cell_type": "markdown", + "id": "0060fcca", + "metadata": {}, + "source": [ + "### Save Model as TorchScript\n", + "This saves an [intermediate representation of the compute graph](https://pytorch.org/tutorials/beginner/saving_loading_models.html#export-load-model-in-torchscript-format), which does not require pickle (or even python). " ] }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 13, "id": "20fedb5d-c59e-4b0b-ba91-3dd15df1f09e", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saved TorchScript Model to ts_housing_model.pt\n" + ] + } + ], "source": [ "scripted = torch.jit.script(mlp)\n", - "scripted.save(\"housing_model.ts\")" + "scripted.save(\"ts_housing_model.pt\")\n", + "print(\"Saved TorchScript Model to ts_housing_model.pt\")" ] }, { @@ -318,22 +391,34 @@ "id": "3101c0fe-65f1-411e-9192-e8a6b585ba0d", "metadata": {}, "source": [ - "### Load and Test Model" + "### Load and Test from Model State" ] }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 14, "id": "7411b00f-88d2-40f5-b716-a26733c968ff", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "loaded_mlp = torch.load(\"housing_model.pt\")" + "loaded_mlp = MLP().to(device)\n", + "loaded_mlp.load_state_dict(torch.load(\"housing_model.pt\", weights_only=True))" ] }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 15, "id": "e226f449-2931-4492-9003-503cdc61f061", "metadata": {}, "outputs": [], @@ -343,48 +428,48 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 16, "id": "d46af47e-db7e-42ee-9bd3-6e7d93850be3", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "tensor([[2.8778],\n", - " [0.6233],\n", - " [3.9021],\n", - " [2.4543],\n", - " [1.0209],\n", - " [1.8093],\n", - " [1.4593],\n", - " [3.2933],\n", - " [2.9263],\n", - " [1.4790]], grad_fn=)" + "tensor([[1.7498],\n", + " [3.0116],\n", + " [1.1925],\n", + " [4.0598],\n", + " [2.0545],\n", + " [2.9072],\n", + " [2.0551],\n", + " [4.6094],\n", + " [1.0068],\n", + " [1.1174]], device='cuda:0', grad_fn=)" ] }, - "execution_count": 14, + "execution_count": 16, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "loaded_mlp(testX)" + "loaded_mlp(testX.to(device))" ] }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 17, "id": "13ae2c0f-1da5-45a4-bf32-ed8b562d7907", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "tensor([2.8380, 0.5740, 5.0000, 2.0430, 1.2680, 1.8380, 1.4340, 2.6830, 1.6100,\n", - " 1.3050])" + "tensor([3.5000, 2.9130, 0.7020, 5.0000, 1.7970, 2.7080, 2.1470, 5.0000, 0.6000,\n", + " 0.8480])" ] }, - "execution_count": 15, + "execution_count": 17, "metadata": {}, "output_type": "execute_result" } @@ -393,19 +478,27 @@ "testY" ] }, + { + "cell_type": "markdown", + "id": "3bcd329d", + "metadata": {}, + "source": [ + "### Load and Test from TorchScript" + ] + }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 18, "id": "422e317f-c9bd-4f76-9463-7af2935d401d", "metadata": {}, "outputs": [], "source": [ - "scripted_mlp = torch.jit.load(\"housing_model.ts\")" + "scripted_mlp = torch.jit.load(\"ts_housing_model.pt\")" ] }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 19, "id": "0cda8ec8-644e-4888-bfa0-b79425ece7c3", "metadata": { "tags": [] @@ -414,17 +507,264 @@ { "data": { "text/plain": [ - "tensor([2.8778, 0.6233, 3.9021, 2.4543, 1.0209, 1.8093, 1.4593, 3.2933, 2.9263,\n", - " 1.4790], grad_fn=)" + "tensor([1.7498, 3.0116, 1.1925, 4.0598, 2.0545, 2.9072, 2.0551, 4.6094, 1.0068,\n", + " 1.1174], device='cuda:0', grad_fn=)" ] }, - "execution_count": 17, + "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "scripted_mlp(testX).flatten()" + "scripted_mlp(testX.to(device)).flatten()" + ] + }, + { + "cell_type": "markdown", + "id": "2a3b64e4", + "metadata": {}, + "source": [ + "### Compile using the Torch JIT Compiler\n", + "This leverages the [Torch-TensorRT inference compiler](https://pytorch.org/TensorRT/) for accelerated inference on GPUs using the `torch.compile` JIT interface under the hood. The compiler stack returns a [boxed-function](http://blog.ezyang.com/2020/09/lets-talk-about-the-pytorch-dispatcher/) that triggers compilation on the first call. \n", + "\n", + "Modules compiled in this fashion are [not serializable with pickle](https://github.com/pytorch/pytorch/issues/101107#issuecomment-1542688089), so we cannot send the compiled model directly to Spark. Instead, we will recompile and cache the model on the executor. " + ] + }, + { + "cell_type": "markdown", + "id": "c613f24e", + "metadata": {}, + "source": [ + "(You may see a warning about modelopt quantization. This is safe to ignore, as [implicit quantization](https://docs.nvidia.com/deeplearning/tensorrt/developer-guide/index.html#intro-quantization) is deprecated in the latest TensorRT. See [this link](https://pytorch.org/TensorRT/tutorials/_rendered_examples/dynamo/vgg16_fp8_ptq.html) for a guide to explicit quantization.)" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "b4aa2523", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING:torch_tensorrt.dynamo.conversion.aten_ops_converters:Unable to import quantization op. Please install modelopt library (https://github.com/NVIDIA/TensorRT-Model-Optimizer?tab=readme-ov-file#installation) to add support for compiling quantized models\n", + "INFO:torch_tensorrt.dynamo.utils:Using Default Torch-TRT Runtime (as requested by user)\n", + "INFO:torch_tensorrt.dynamo.utils:Device not specified, using Torch default current device - cuda:0. If this is incorrect, please specify an input device, via the device keyword.\n", + "INFO:torch_tensorrt.dynamo.utils:Compilation Settings: CompilationSettings(enabled_precisions={}, debug=False, workspace_size=0, min_block_size=5, torch_executed_ops=set(), pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, assume_dynamic_shape_support=False, sparse_weights=False, refit=False, engine_capability=, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912, dryrun=False, hardware_compatible=False, timing_cache_path='/tmp/timing_cache.bin')\n", + "\n", + "WARNING:torch_tensorrt.dynamo._compiler:Node _param_constant1 of op type get_attr does not have metadata. This could sometimes lead to undefined behavior.\n", + "WARNING:torch_tensorrt.dynamo._compiler:Some nodes do not have metadata (shape and dtype information). This could lead to problems sometimes if the graph has PyTorch and TensorRT segments.\n", + "INFO:torch_tensorrt.dynamo._compiler:Partitioning the graph via the fast partitioner\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init CUDA: CPU +1, GPU +0, now: CPU 586, GPU 1112 (MiB)\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init builder kernel library: CPU +1635, GPU +288, now: CPU 2368, GPU 1400 (MiB)\n", + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/torch_tensorrt/dynamo/conversion/impl/activation/base.py:40: DeprecationWarning: Use Deprecated in TensorRT 10.1. Superseded by explicit quantization. instead.\n", + " if input_val.dynamic_range is not None and dyn_range_fn is not None:\n", + "\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT INetwork construction elapsed time: 0:00:00.003844\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Global timing cache in use. Profiling results in this builder pass will be stored.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Detected 1 inputs and 1 output network tensors.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Host Persistent Memory: 22240\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Device Persistent Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Scratch Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Started assigning block shifts. This will take 10 steps to complete.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Algorithm ShiftNTopDown took 0.156445ms to assign 4 blocks to 10 nodes requiring 7168 bytes.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Activation Memory: 6656\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Weights Memory: 11648\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Engine generation completed in 0.0225783 seconds.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1 MiB\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage during Engine building and serialization: CPU: 3966 MiB\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:Build TRT engine elapsed time: 0:00:00.026248\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT Engine uses: 665972 bytes of Memory\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 26 bytes of code generator cache.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 176 timing cache entries\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "tensor([[1.7498],\n", + " [3.0116],\n", + " [1.1925],\n", + " [4.0598],\n", + " [2.0545],\n", + " [2.9072],\n", + " [2.0551],\n", + " [4.6094],\n", + " [1.0068],\n", + " [1.1174]], device='cuda:0')\n" + ] + } + ], + "source": [ + "import torch_tensorrt as trt\n", + "\n", + "inputs_bs1 = torch.randn((10, 8), dtype=torch.float).to(\"cuda\")\n", + "# This indicates dimension 0 of inputs_bs1 is dynamic whose range of values is [1, 50]. No recompilation will happen when the batch size changes.\n", + "torch._dynamo.mark_dynamic(inputs_bs1, 0, min=1, max=50)\n", + "trt_model = trt.compile(\n", + " loaded_mlp,\n", + " ir=\"torch_compile\",\n", + " inputs=inputs_bs1,\n", + " enabled_precisions={torch.float},\n", + ")\n", + "\n", + "stream = torch.cuda.Stream()\n", + "with torch.no_grad(), torch.cuda.stream(stream):\n", + " testX = testX.to(device)\n", + " print(trt_model(testX))" + ] + }, + { + "cell_type": "markdown", + "id": "d2c55e07", + "metadata": {}, + "source": [ + "### Compile using the Torch-TensorRT AOT Compiler\n", + "Alternatively, use the Torch-TensorRT Dynamo backend for Ahead-of-Time (AOT) compilation to eagerly optimize the model in an explicit compilation phase. We first export the model to produce a traced graph representing the Tensor computation in an AOT fashion, which produces a `ExportedProgram` object which can be [serialized and reloaded](https://pytorch.org/TensorRT/user_guide/saving_models.html). We can then compile this IR using the Torch-TensorRT AOT compiler for inference. \n", + "\n", + "[Read the docs](https://pytorch.org/TensorRT/user_guide/torch_tensorrt_explained.html) for more information on JIT vs AOT compilation." + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "b6b5c112", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:torch_tensorrt.dynamo._compiler:Compilation Settings: CompilationSettings(enabled_precisions={}, debug=False, workspace_size=0, min_block_size=5, torch_executed_ops=set(), pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, assume_dynamic_shape_support=False, sparse_weights=False, refit=False, engine_capability=, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912, dryrun=False, hardware_compatible=False, timing_cache_path='/tmp/timing_cache.bin')\n", + "\n", + "INFO:torch_tensorrt.dynamo._compiler:Partitioning the graph via the fast partitioner\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 762, GPU 1114 (MiB)\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init builder kernel library: CPU +1633, GPU +286, now: CPU 2395, GPU 1400 (MiB)\n", + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/torch_tensorrt/dynamo/conversion/impl/activation/base.py:40: DeprecationWarning: Use Deprecated in TensorRT 10.1. Superseded by explicit quantization. instead.\n", + " if input_val.dynamic_range is not None and dyn_range_fn is not None:\n", + "\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT INetwork construction elapsed time: 0:00:00.002832\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Global timing cache in use. Profiling results in this builder pass will be stored.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Detected 1 inputs and 1 output network tensors.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Host Persistent Memory: 22240\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Device Persistent Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Scratch Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Started assigning block shifts. This will take 10 steps to complete.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Algorithm ShiftNTopDown took 0.120805ms to assign 4 blocks to 10 nodes requiring 7168 bytes.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Activation Memory: 6656\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Weights Memory: 11648\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Engine generation completed in 0.014855 seconds.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1 MiB\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage during Engine building and serialization: CPU: 3989 MiB\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:Build TRT engine elapsed time: 0:00:00.016879\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT Engine uses: 666804 bytes of Memory\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 26 bytes of code generator cache.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 176 timing cache entries\n", + "INFO:torch_tensorrt.dynamo.utils:Using Default Torch-TRT Runtime (as requested by user)\n", + "INFO:torch_tensorrt.dynamo.utils:Device not specified, using Torch default current device - cuda:0. If this is incorrect, please specify an input device, via the device keyword.\n", + "INFO:torch_tensorrt.dynamo.utils:Compilation Settings: CompilationSettings(enabled_precisions={}, debug=False, workspace_size=0, min_block_size=5, torch_executed_ops=set(), pass_through_build_failures=False, max_aux_streams=None, version_compatible=False, optimization_level=None, use_python_runtime=False, truncate_double=False, use_fast_partitioner=True, enable_experimental_decompositions=False, device=Device(type=DeviceType.GPU, gpu_id=0), require_full_compilation=False, disable_tf32=False, assume_dynamic_shape_support=False, sparse_weights=False, refit=False, engine_capability=, num_avg_timing_iters=1, dla_sram_size=1048576, dla_local_dram_size=1073741824, dla_global_dram_size=536870912, dryrun=False, hardware_compatible=False, timing_cache_path='/tmp/timing_cache.bin')\n", + "\n", + "WARNING:torch_tensorrt.dynamo._compiler:Node _param_constant1 of op type get_attr does not have metadata. This could sometimes lead to undefined behavior.\n", + "WARNING:torch_tensorrt.dynamo._compiler:Some nodes do not have metadata (shape and dtype information). This could lead to problems sometimes if the graph has PyTorch and TensorRT segments.\n", + "INFO:torch_tensorrt.dynamo._compiler:Partitioning the graph via the fast partitioner\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:The logger passed into createInferBuilder differs from one already provided for an existing builder, runtime, or refitter. Uses of the global logger, returned by nvinfer1::getLogger(), will return the existing value.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 764, GPU 1136 (MiB)\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageChange] Init builder kernel library: CPU +1632, GPU +286, now: CPU 2396, GPU 1422 (MiB)\n", + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/torch_tensorrt/dynamo/conversion/impl/activation/base.py:40: DeprecationWarning: Use Deprecated in TensorRT 10.1. Superseded by explicit quantization. instead.\n", + " if input_val.dynamic_range is not None and dyn_range_fn is not None:\n", + "\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT INetwork construction elapsed time: 0:00:00.002990\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Global timing cache in use. Profiling results in this builder pass will be stored.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Detected 1 inputs and 1 output network tensors.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Host Persistent Memory: 22240\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Device Persistent Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Scratch Memory: 0\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Started assigning block shifts. This will take 10 steps to complete.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[BlockAssignment] Algorithm ShiftNTopDown took 0.243798ms to assign 4 blocks to 10 nodes requiring 7168 bytes.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Activation Memory: 6656\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Total Weights Memory: 11648\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Engine generation completed in 0.0158591 seconds.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 1 MiB\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:[MemUsageStats] Peak memory usage during Engine building and serialization: CPU: 3991 MiB\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:Build TRT engine elapsed time: 0:00:00.017873\n", + "INFO:torch_tensorrt.dynamo.conversion._TRTInterpreter:TRT Engine uses: 665972 bytes of Memory\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 26 bytes of code generator cache.\n", + "INFO:torch_tensorrt [TensorRT Conversion Context]:Serialized 176 timing cache entries\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "tensor([[1.7498],\n", + " [3.0116],\n", + " [1.1925],\n", + " [4.0598],\n", + " [2.0545],\n", + " [2.9072],\n", + " [2.0551],\n", + " [4.6094],\n", + " [1.0068],\n", + " [1.1174]], device='cuda:0')\n" + ] + } + ], + "source": [ + "# Preparing the inputs for batch_size = 50. \n", + "inputs = (torch.randn((10, 8), dtype=torch.float).cuda(),)\n", + "\n", + "# Produce traced graph in the ExportedProgram format\n", + "exp_program = trt.dynamo.trace(loaded_mlp, inputs)\n", + "# Compile the traced graph to produce an optimized module\n", + "trt_gm = trt.dynamo.compile(exp_program, inputs=inputs, device='cuda:0')\n", + "\n", + "stream = torch.cuda.Stream()\n", + "with torch.no_grad(), torch.cuda.stream(stream):\n", + " testX = testX.to(device)\n", + " print(trt_model(testX))" + ] + }, + { + "cell_type": "markdown", + "id": "b4fef57d", + "metadata": {}, + "source": [ + "We can save the compiled model using `torch_tensorrt.save`. Unfortunately, serializing the model to be reloaded at a later date currently only supports *static inputs*." + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "dabc91a4", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/torch_tensorrt/dynamo/_exporter.py:364: UserWarning: Attempted to insert a get_attr Node with no underlying reference in the owning GraphModule! Call GraphModule.add_submodule to add the necessary submodule, GraphModule.add_parameter to add the necessary Parameter, or nn.Module.register_buffer to add the necessary buffer\n", + " engine_node = gm.graph.get_attr(engine_name)\n", + "\n", + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/torch/fx/graph.py:1545: UserWarning: Node _run_on_acc_0_engine target _run_on_acc_0_engine _run_on_acc_0_engine of does not reference an nn.Module, nn.Parameter, or buffer, which is what 'get_attr' Nodes typically target\n", + " warnings.warn(f'Node {node} target {node.target} {atom} of {seen_qualname} does '\n", + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saved AOT compiled TensorRT model to trt_model_aot.ep\n" + ] + } + ], + "source": [ + "with torch.cuda.stream(stream):\n", + " trt.save(trt_gm, \"trt_model_aot.ep\", inputs=[torch.randn((10, 8), dtype=torch.float).to(\"cuda\")])\n", + " print(\"Saved AOT compiled TensorRT model to trt_model_aot.ep\")" ] }, { @@ -437,7 +777,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 23, "id": "32e11813-cf75-448e-a46e-f210cc7f52ba", "metadata": {}, "outputs": [], @@ -455,17 +795,17 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 24, "id": "dc7da567-65df-4895-a867-0be05de27ee0", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "" + "" ] }, - "execution_count": 19, + "execution_count": 24, "metadata": {}, "output_type": "execute_result" } @@ -476,7 +816,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 25, "id": "e3340e01-b3bc-4cce-bb21-890517e1bcd5", "metadata": {}, "outputs": [], @@ -486,7 +826,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 26, "id": "dea2ecd8-34e2-4ed5-8bd6-c9d56c951eeb", "metadata": {}, "outputs": [], @@ -520,7 +860,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 27, "id": "f52f4640-e190-413e-8e01-d67492408f97", "metadata": {}, "outputs": [], @@ -531,7 +871,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 28, "id": "e2179934-6ae0-4d58-ae2c-d82f90d48074", "metadata": {}, "outputs": [ @@ -558,7 +898,7 @@ " 2.0530])]" ] }, - "execution_count": 23, + "execution_count": 28, "metadata": {}, "output_type": "execute_result" } @@ -569,7 +909,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 29, "id": "7fc3383b-fb1c-4daf-9844-88df7abf799d", "metadata": {}, "outputs": [], @@ -592,18 +932,18 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 30, "id": "25e9de54-a8da-46da-ba89-177a75227420", "metadata": {}, "outputs": [], "source": [ "# Initialize the MLP\n", - "mlp2 = MLP2()" + "mlp2 = MLP2().to(device)" ] }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 31, "id": "631116aa-e496-4125-9970-14aeb816c106", "metadata": {}, "outputs": [], @@ -615,7 +955,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 32, "id": "509d0581-5911-4f21-b0c8-b94523f66dd2", "metadata": { "scrolled": true, @@ -705,6 +1045,7 @@ "\n", " # Get and prepare inputs\n", " a,b,c,d,e,f,g,h,targets = data\n", + " a,b,c,d,e,f,g,h,targets = a.to(device),b.to(device),c.to(device),d.to(device),e.to(device),f.to(device),g.to(device),h.to(device),targets.to(device)\n", " targets = targets.reshape((targets.shape[0], 1))\n", "\n", " # Zero the gradients\n", @@ -738,28 +1079,54 @@ "id": "5029a35d-8fbd-4a11-b3b0-55bcc0a072dd", "metadata": {}, "source": [ - "### Save Model" + "### Save Model State Dict" ] }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 33, "id": "ca720ac4-8b4e-489b-844f-d54dd0659755", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saved PyTorch Model State to housing_model2.pt\n" + ] + } + ], + "source": [ + "torch.save(mlp2.state_dict(), \"housing_model2.pt\")\n", + "print(\"Saved PyTorch Model State to housing_model2.pt\")" + ] + }, + { + "cell_type": "markdown", + "id": "7f677429", + "metadata": {}, "source": [ - "torch.save(mlp2, \"housing_model2.pt\")" + "### Save Model as TorchScript" ] }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 34, "id": "cdcced78-62ee-45fa-b334-6f73a2b21d32", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saved TorchScript Model to ts_housing_model2.pt\n" + ] + } + ], "source": [ "scripted = torch.jit.script(mlp2)\n", - "scripted.save(\"housing_model2.ts\")" + "scripted.save(\"ts_housing_model2.pt\")\n", + "print(\"Saved TorchScript Model to ts_housing_model2.pt\")" ] }, { @@ -767,32 +1134,45 @@ "id": "8ecb33c2-2e3f-487c-8b12-c4e8f13a67a2", "metadata": {}, "source": [ - "### Load and Test Model" + "### Load and Test from Model State" ] }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 35, "id": "c54b12f2-9981-477b-8c21-a652a1736fc9", "metadata": {}, "outputs": [], "source": [ - "a,b,c,d,e,f,g,h,targets = next(iter(trainloader2))" + "a,b,c,d,e,f,g,h,targets = next(iter(trainloader2))\n", + "a,b,c,d,e,f,g,h,targets = a.to(device), b.to(device), c.to(device), d.to(device), e.to(device), f.to(device), g.to(device), h.to(device), targets.to(device)" ] }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 36, "id": "31b2fa69-9a8a-4409-8652-23c547536e50", "metadata": {}, - "outputs": [], + "outputs": [ + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 36, + "metadata": {}, + "output_type": "execute_result" + } + ], "source": [ - "loaded_mlp2 = torch.load(\"housing_model2.pt\")" + "loaded_mlp2 = MLP2().to(device)\n", + "loaded_mlp2.load_state_dict(torch.load(\"housing_model2.pt\", weights_only=True))" ] }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 37, "id": "80a21d52-ea98-4c74-98e8-1e088cdfa742", "metadata": {}, "outputs": [ @@ -808,10 +1188,10 @@ " [1.4593],\n", " [3.2933],\n", " [2.9263],\n", - " [1.4790]], grad_fn=)" + " [1.4790]], device='cuda:0', grad_fn=)" ] }, - "execution_count": 32, + "execution_count": 37, "metadata": {}, "output_type": "execute_result" } @@ -822,7 +1202,7 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 38, "id": "90d29fb0-5923-4684-8aa6-62618e8f1ef6", "metadata": {}, "outputs": [ @@ -838,19 +1218,27 @@ "print(signature(loaded_mlp2.forward))" ] }, + { + "cell_type": "markdown", + "id": "de84bc12", + "metadata": {}, + "source": [ + "### Load and Test from TorchScript" + ] + }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 39, "id": "165772b2-8277-4b6e-a178-c99ea2a031fa", "metadata": {}, "outputs": [], "source": [ - "scripted_mlp2 = torch.jit.load(\"housing_model2.ts\")" + "scripted_mlp2 = torch.jit.load(\"ts_housing_model2.pt\")" ] }, { "cell_type": "code", - "execution_count": 35, + "execution_count": 40, "id": "e53d927e-fa6d-419d-b570-8ca9b0756812", "metadata": {}, "outputs": [ @@ -866,10 +1254,10 @@ " [1.4593],\n", " [3.2933],\n", " [2.9263],\n", - " [1.4790]], grad_fn=)" + " [1.4790]], device='cuda:0', grad_fn=)" ] }, - "execution_count": 35, + "execution_count": 40, "metadata": {}, "output_type": "execute_result" } @@ -896,7 +1284,7 @@ }, { "cell_type": "code", - "execution_count": 36, + "execution_count": 41, "id": "cf35da14-61a3-4e7b-9d4f-086bf5e931b3", "metadata": {}, "outputs": [], @@ -906,7 +1294,7 @@ }, { "cell_type": "code", - "execution_count": 37, + "execution_count": 42, "id": "95148019-ea95-40e5-a529-fcdb9a06f928", "metadata": {}, "outputs": [], @@ -916,7 +1304,7 @@ }, { "cell_type": "code", - "execution_count": 38, + "execution_count": 43, "id": "f82d957c-6747-4408-aac8-45305afbfe5e", "metadata": {}, "outputs": [ @@ -1079,8 +1467,8 @@ "" ], "text/plain": [ - " MedInc HouseAge AveRooms AveBedrms Population AveOccup \n", - "0 2.344766 0.982143 0.628559 -0.153758 -0.974429 -0.049597 \\\n", + " MedInc HouseAge AveRooms AveBedrms Population AveOccup \\\n", + "0 2.344766 0.982143 0.628559 -0.153758 -0.974429 -0.049597 \n", "1 2.332238 -0.607019 0.327041 -0.263336 0.861439 -0.092512 \n", "2 1.782699 1.856182 1.155620 -0.049016 -0.820777 -0.025843 \n", "3 0.932967 1.856182 0.156966 -0.049833 -0.766028 -0.050329 \n", @@ -1108,7 +1496,7 @@ "[20640 rows x 8 columns]" ] }, - "execution_count": 38, + "execution_count": 43, "metadata": {}, "output_type": "execute_result" } @@ -1120,7 +1508,7 @@ }, { "cell_type": "code", - "execution_count": 39, + "execution_count": 44, "id": "5ba338cd-76d2-46bd-baf5-7d18a339a449", "metadata": {}, "outputs": [], @@ -1130,7 +1518,7 @@ }, { "cell_type": "code", - "execution_count": 40, + "execution_count": 45, "id": "224b5036-d2ed-4edf-975f-66127862343d", "metadata": {}, "outputs": [ @@ -1140,7 +1528,7 @@ "dict_keys(['MedInc', 'HouseAge', 'AveRooms', 'AveBedrms', 'Population', 'AveOccup', 'Latitude', 'Longitude'])" ] }, - "execution_count": 40, + "execution_count": 45, "metadata": {}, "output_type": "execute_result" } @@ -1151,7 +1539,7 @@ }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 46, "id": "0b32ea98-a7f1-4011-a067-700377f1717f", "metadata": {}, "outputs": [ @@ -1169,7 +1557,7 @@ "dtype: object" ] }, - "execution_count": 41, + "execution_count": 46, "metadata": {}, "output_type": "execute_result" } @@ -1180,7 +1568,19 @@ }, { "cell_type": "code", - "execution_count": 42, + "execution_count": 47, + "id": "e630236c", + "metadata": {}, + "outputs": [], + "source": [ + "from pyspark.sql.types import *\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf" + ] + }, + { + "cell_type": "code", + "execution_count": 48, "id": "6388cce9-6469-4f5a-898a-1a0b74eec438", "metadata": {}, "outputs": [ @@ -1188,52 +1588,86 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 0:> (0 + 1) / 1]\r" + "24/10/08 00:36:22 WARN Utils: Your hostname, cb4ae00-lcedt resolves to a loopback address: 127.0.1.1; using 10.110.47.100 instead (on interface eno1)\n", + "24/10/08 00:36:22 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address\n", + "Setting default log level to \"WARN\".\n", + "To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).\n", + "24/10/08 00:36:22 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\n" ] - }, + } + ], + "source": [ + "import os\n", + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" + ] + }, + { + "cell_type": "code", + "execution_count": 49, + "id": "881afee9", + "metadata": {}, + "outputs": [ { - "name": "stdout", + "name": "stderr", "output_type": "stream", "text": [ - "+------------+----------+------------+------------+-----------+------------+---------+----------+\n", - "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude|\n", - "+------------+----------+------------+------------+-----------+------------+---------+----------+\n", - "| 2.344766| 0.9821427| 0.62855947| -0.15375753|-0.97442853|-0.049596533|1.0525488|-1.3278369|\n", - "| 2.3322382|-0.6070189| 0.32704142| -0.26333576| 0.8614389| -0.09251223|1.0431849|-1.3228445|\n", - "| 1.7826993| 1.8561815| 1.1556205|-0.049016476|-0.82077736|-0.025842525| 1.038502|-1.3328254|\n", - "| 0.93296736| 1.8561815| 0.15696616|-0.049833003|-0.76602805|-0.050329294| 1.038502|-1.3378178|\n", - "|-0.012881001| 1.8561815| 0.34471077|-0.032905966| -0.7598467| -0.08561575| 1.038502|-1.3378178|\n", - "| 0.087446585| 1.8561815| -0.26972958| 0.014669393|-0.89407074|-0.089618415| 1.038502|-1.3378178|\n", - "| -0.11136628| 1.8561815| -0.20091766| -0.30663314| -0.2927116| -0.09072491|1.0338209|-1.3378178|\n", - "| -0.39513668| 1.8561815| -0.255232|-0.073541574|-0.23707923| -0.12347647|1.0338209|-1.3378178|\n", - "| -0.94235915| 1.0616008| -0.45870265| 0.04425391|-0.19380963| -0.10049919|1.0338209|-1.3428102|\n", - "| -0.09446957| 1.8561815| -0.18528317| -0.22468716| 0.1108437| -0.08650142|1.0338209|-1.3378178|\n", - "| -0.35139465| 1.8561815| 0.019648364|-0.036026947|-0.45519334| -0.07769971| 1.038502|-1.3428102|\n", - "| -0.31591678| 1.8561815| -0.26535577| -0.15225175| 0.06934021| -0.09836594| 1.038502|-1.3428102|\n", - "| -0.41882363| 1.8561815|-0.042985205| -0.17694615|-0.28917935| -0.06975886| 1.038502|-1.3428102|\n", - "| -0.6301118| 1.8561815| -0.5775806| 0.002165105| -0.9541183| -0.10474847|1.0338209|-1.3428102|\n", - "| -1.0285273| 1.8561815| -0.471319| -0.1835785|-0.18851131| -0.10743675| 1.038502|-1.3428102|\n", - "| -0.9188827| 1.6972654| -0.4795964| -0.05213217|-0.64328367|-0.041451186| 1.038502|-1.3428102|\n", - "| -0.576737| 1.8561815| 0.20636784| -0.10199789| -0.5585106| -0.06498151| 1.038502|-1.3477987|\n", - "| -0.92140937| 1.8561815| -0.5562374| -0.27364123| -0.6865533| -0.08974189| 1.038502|-1.3477987|\n", - "| -0.98936474| 1.6972654|-0.034486752|-0.022697322| -0.3845491| -0.06815911|1.0338209|-1.3428102|\n", - "| -0.6671161| 1.8561815| 0.014734507|-0.027513746| -0.649465|-0.054070402|1.0338209|-1.3477987|\n", - "+------------+----------+------------+------------+-----------+------------+---------+----------+\n", - "only showing top 20 rows\n", - "\n" + "WARNING:py.warnings:/home/rishic/anaconda3/envs/spark-dl-torch/lib/python3.11/site-packages/pyspark/sql/pandas/serializers.py:224: DeprecationWarning: is_categorical_dtype is deprecated and will be removed in a future version. Use isinstance(dtype, pd.CategoricalDtype) instead\n", + " if is_categorical_dtype(series.dtype):\n", + "\n", + " \r" ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ - " \r" + "+------------+------------+-----------+------------+-----------+------------+----------+------------+\n", + "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+\n", + "| 0.20909257| -1.1632254| 0.38946992| 0.04609274| -0.9806099| -0.07099328|0.61245227|-0.020113053|\n", + "|-0.098627955| 0.34647804| 0.27216315| -0.0129226| -0.6953838| -0.05380849| 1.0665938| -1.2479742|\n", + "| -0.66006273| 1.0616008|-0.55292207| -0.48945764|-0.13641118| 0.028952759| 1.1040496| -1.3827378|\n", + "| 0.08218294| 0.5848523|-0.13912922| -0.14707813|-0.19116047| -0.07136432|0.96827507| -1.3028787|\n", + "| 0.0784456| -1.4810578| 0.57265776| 0.32067496| 1.0345173|-0.024157424| 1.4411427| -0.52423614|\n", + "| -0.82318723| -0.36864465| 0.07829511| -0.1808107|-0.67242444|-0.061470542| 1.9374212| -1.0083897|\n", + "| 0.59671736| 0.5848523| 0.19346413| -0.1371872|-0.19645879| 0.009964322|0.96827507| -1.2928978|\n", + "| -0.9612035| -1.5605159|-0.56329846| 0.027148023|-0.71127874| -0.08471591| 0.5328614| -0.13990337|\n", + "| -0.74344087| -1.2426835| 0.27282518| 0.4037246| -0.9841421| -0.05610115| 1.2257773| -0.42940006|\n", + "| 0.9784464| -0.2891866| 0.24374022| -0.24670053| 0.28922042| -0.01102468| 1.1087307| -1.2280084|\n", + "| -0.5070446| -1.0043093|-0.78254056|0.0122275995| 2.8465424|-0.060435444| 0.8980464| -1.2080427|\n", + "| -0.18690155| 1.2205169|0.015323491| 0.12183313|-0.41015765| 0.04452552| 1.010412| -1.3228445|\n", + "| -1.2551856| 1.6178073| -0.3341509|-0.060125165| -0.7554314| -0.08777025| 1.0291398| -1.3477987|\n", + "| 4.9607058| -1.9578062| 1.4854684| -0.03948475| 2.1833694|0.0029250523| 1.024457| -1.1581304|\n", + "| 0.73652315| -1.6399739| 0.7913185| -0.05238397| 1.67738| 0.01944797| 1.0993668| -1.1331724|\n", + "| -0.505834| 0.18756187|-0.47093546| -0.24297306|-0.60619545| -0.10791535| 0.977639| -1.2879055|\n", + "| -0.88477343|-0.050812364| -0.6318951| -0.15244243| -0.5258376| -0.15618815| 0.9823201| -1.2879055|\n", + "| -0.42840376| 0.9821427| -0.2266495| -0.36083496| -0.6883194| -0.08552282| 0.5328614| -0.12493005|\n", + "| 0.9369153| -1.4810578| 0.6722208|-0.121177554| 0.3996021| 0.01291408| 1.1040496| -1.1082181|\n", + "| -0.80702734| -0.92485124|-0.26602685| -0.1560743| 1.4398388| -0.09314839|0.55627036| -0.09498342|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+\n", + "only showing top 20 rows\n", + "\n" ] } ], "source": [ - "from pyspark.sql.types import *\n", - "\n", "# Spark is somehow auto-converting Pandas float32 to DoubleType(), so forcing FloatType()\n", "schema = StructType([\n", "StructField(\"MedInc\",FloatType(),True),\n", @@ -1246,13 +1680,13 @@ "StructField(\"Longitude\",FloatType(),True)\n", "])\n", "\n", - "df = spark.createDataFrame(pdf, schema=schema)\n", + "df = spark.createDataFrame(pdf, schema=schema).repartition(8)\n", "df.show(truncate=12)" ] }, { "cell_type": "code", - "execution_count": 43, + "execution_count": 50, "id": "7b33d367-fbf9-4918-b755-5447125547c4", "metadata": {}, "outputs": [ @@ -1262,7 +1696,7 @@ "StructType([StructField('MedInc', FloatType(), True), StructField('HouseAge', FloatType(), True), StructField('AveRooms', FloatType(), True), StructField('AveBedrms', FloatType(), True), StructField('Population', FloatType(), True), StructField('AveOccup', FloatType(), True), StructField('Latitude', FloatType(), True), StructField('Longitude', FloatType(), True)])" ] }, - "execution_count": 43, + "execution_count": 50, "metadata": {}, "output_type": "execute_result" } @@ -1281,18 +1715,10 @@ }, { "cell_type": "code", - "execution_count": 44, + "execution_count": 51, "id": "751bff7a-b687-4184-b3fa-b5f5b46ef5d1", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - } - ], + "outputs": [], "source": [ "df.write.mode(\"overwrite\").parquet(\"california_housing\")" ] @@ -1308,11 +1734,12 @@ }, { "cell_type": "code", - "execution_count": 45, + "execution_count": 52, "id": "986d1a97-ea84-4707-b94a-78498780c47c", "metadata": {}, "outputs": [], "source": [ + "import os\n", "from pyspark.ml.functions import predict_batch_udf\n", "from pyspark.sql.functions import array, struct, col\n", "from pyspark.sql.types import ArrayType, FloatType" @@ -1320,7 +1747,7 @@ }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 53, "id": "1e40c266-24de-454d-a776-f3716ba50e90", "metadata": {}, "outputs": [], @@ -1330,7 +1757,7 @@ }, { "cell_type": "code", - "execution_count": 47, + "execution_count": 54, "id": "ac802fb6-f159-4776-b55d-b9c421e8c57e", "metadata": {}, "outputs": [ @@ -1347,7 +1774,7 @@ " 'Longitude']" ] }, - "execution_count": 47, + "execution_count": 54, "metadata": {}, "output_type": "execute_result" } @@ -1359,7 +1786,7 @@ }, { "cell_type": "code", - "execution_count": 48, + "execution_count": 55, "id": "4b8de001-e791-4a91-bd6f-c80bdf1c4472", "metadata": {}, "outputs": [ @@ -1367,30 +1794,30 @@ "name": "stdout", "output_type": "stream", "text": [ - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+\n", - "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude|\n", - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+\n", - "| -1.669445|-0.20972852| -1.1155425| -0.17418891| 0.2070965| -0.13437383| 0.5094524| -0.08001011|\n", - "| -0.8564016| -1.5605159| -0.53141737| -0.02190494| -0.66006166| -0.12997007| 0.5141335| -0.08001011|\n", - "| 0.73173314|-0.76593506| 0.67250663| -0.10979619| 0.14175056| -0.02296524| 0.5141335| -0.07002536|\n", - "|-0.44887984| -1.719432| 0.14690235| -0.009905009| -0.06664997| 0.01090982|0.50476956|-0.075017735|\n", - "|-0.96920437| -1.0837674| 0.058284093| 0.07351444| -0.20793848| -0.011342554| 0.5094524| -0.08001011|\n", - "| -1.0906925| 0.26701993| -0.72011936| -0.2243873| -0.6574125| 0.021105729|0.49072453| -0.08001011|\n", - "|-0.90240705| 0.5848523| 0.24784403| 0.08091579| -0.5205393| 0.043980353| 0.4954056|-0.060044423|\n", - "| -1.0712692|0.028645715| -0.38059494| 0.062626354| -0.5205393| -0.005065064|0.49072453| -0.09498342|\n", - "|0.026544658| -0.686477| 0.06193017| -0.20868859| 0.03401808| 0.017589226| 0.4766795|-0.050059676|\n", - "|-0.24211854| 0.34647804| 0.2122066| -0.11910806| -0.71127874| 0.047426607|0.47199664|-0.030093992|\n", - "| -1.1632799| -1.7988901| -0.3091| -0.16322336| -0.0719483| 0.01691941|0.46263272|-0.040078737|\n", - "| -0.4509853| 0.8232265| -0.10236703| -0.053904843| -0.6953838| -0.03862959|0.46731555| -0.05505205|\n", - "| -0.5102555|-0.76593506| 0.26753396| 0.023170268| -0.14877392|-0.0061120046| 0.4766795|-0.084998675|\n", - "| -1.1635431| 0.18756187| -0.59673625| -0.023986263| -0.71481097| -0.014560648|0.46263272| -0.05505205|\n", - "|-0.15400293| 0.5053942| 0.33031902|-0.0012220128| -0.35275918| 0.0010038698|0.46263272| -0.08001011|\n", - "|-0.27733323| 0.18756187|-0.010031368| -0.13782738| -0.32450148| 0.059288267| 0.4485877| -0.08001011|\n", - "| -0.8956694| 0.18756187|-0.043201063| -0.12283955|-0.087843254| 0.011250444|0.49072453| -0.11494911|\n", - "|-0.57936895|-0.13027044| 0.15416715| 0.08602174| -0.6353362| 0.026508931| 0.4766795| -0.10496436|\n", - "| -0.8548752| 0.1081038|-0.061331607| -0.26316243| -0.36865416| 0.0066948985|0.47199664| -0.12493005|\n", - "| -1.1061679| 0.42593613| 0.053027753| -0.12104499| -0.2132368| -0.015556259| 0.4766795| -0.11494911|\n", - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+\n", + "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+\n", + "| 0.20909257| -1.1632254| 0.38946992| 0.04609274| -0.9806099| -0.07099328|0.61245227|-0.020113053|\n", + "|-0.098627955| 0.34647804| 0.27216315| -0.0129226| -0.6953838| -0.05380849| 1.0665938| -1.2479742|\n", + "| -0.66006273| 1.0616008|-0.55292207| -0.48945764|-0.13641118| 0.028952759| 1.1040496| -1.3827378|\n", + "| 0.08218294| 0.5848523|-0.13912922| -0.14707813|-0.19116047| -0.07136432|0.96827507| -1.3028787|\n", + "| 0.0784456| -1.4810578| 0.57265776| 0.32067496| 1.0345173|-0.024157424| 1.4411427| -0.52423614|\n", + "| -0.82318723| -0.36864465| 0.07829511| -0.1808107|-0.67242444|-0.061470542| 1.9374212| -1.0083897|\n", + "| 0.59671736| 0.5848523| 0.19346413| -0.1371872|-0.19645879| 0.009964322|0.96827507| -1.2928978|\n", + "| -0.9612035| -1.5605159|-0.56329846| 0.027148023|-0.71127874| -0.08471591| 0.5328614| -0.13990337|\n", + "| -0.74344087| -1.2426835| 0.27282518| 0.4037246| -0.9841421| -0.05610115| 1.2257773| -0.42940006|\n", + "| 0.9784464| -0.2891866| 0.24374022| -0.24670053| 0.28922042| -0.01102468| 1.1087307| -1.2280084|\n", + "| -0.5070446| -1.0043093|-0.78254056|0.0122275995| 2.8465424|-0.060435444| 0.8980464| -1.2080427|\n", + "| -0.18690155| 1.2205169|0.015323491| 0.12183313|-0.41015765| 0.04452552| 1.010412| -1.3228445|\n", + "| -1.2551856| 1.6178073| -0.3341509|-0.060125165| -0.7554314| -0.08777025| 1.0291398| -1.3477987|\n", + "| 4.9607058| -1.9578062| 1.4854684| -0.03948475| 2.1833694|0.0029250523| 1.024457| -1.1581304|\n", + "| 0.73652315| -1.6399739| 0.7913185| -0.05238397| 1.67738| 0.01944797| 1.0993668| -1.1331724|\n", + "| -0.505834| 0.18756187|-0.47093546| -0.24297306|-0.60619545| -0.10791535| 0.977639| -1.2879055|\n", + "| -0.88477343|-0.050812364| -0.6318951| -0.15244243| -0.5258376| -0.15618815| 0.9823201| -1.2879055|\n", + "| -0.42840376| 0.9821427| -0.2266495| -0.36083496| -0.6883194| -0.08552282| 0.5328614| -0.12493005|\n", + "| 0.9369153| -1.4810578| 0.6722208|-0.121177554| 0.3996021| 0.01291408| 1.1040496| -1.1082181|\n", + "| -0.80702734| -0.92485124|-0.26602685| -0.1560743| 1.4398388| -0.09314839|0.55627036| -0.09498342|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+\n", "only showing top 20 rows\n", "\n" ] @@ -1405,46 +1832,75 @@ "id": "650dda22-31b7-419b-96d4-387a036f3b07", "metadata": {}, "source": [ - "### Using TorchScript Model (single input)" + "### Using TensorRT Model (single input)" ] }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 56, "id": "3d608e2f-66a8-44b5-9cde-5f7837bf4247", "metadata": {}, "outputs": [], "source": [ "# get absolute path to model\n", - "model_dir = \"{}/housing_model.ts\".format(os.getcwd())" + "model_dir = \"{}/housing_model.pt\".format(os.getcwd())" ] }, { "cell_type": "code", - "execution_count": 50, + "execution_count": 57, "id": "a2f45f5d-c941-4197-a274-1eec2af3fca4", "metadata": {}, "outputs": [], "source": [ "def predict_batch_fn():\n", " import torch\n", + " import torch_tensorrt as trt\n", + "\n", " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", - " print(\"Using {} device\".format(device))\n", - " \n", - " scripted_mlp = torch.jit.load(model_dir)\n", - " scripted_mlp.to(device)\n", + " if device != \"cuda\":\n", + " raise ValueError(\"This function uses the TensorRT model which requires a GPU device\")\n", + "\n", + " # Define model\n", + " class MLP(nn.Module):\n", + " def __init__(self):\n", + " super().__init__()\n", + " self.layers = nn.Sequential(\n", + " nn.Linear(8, 64),\n", + " nn.ReLU(),\n", + " nn.Linear(64, 32),\n", + " nn.ReLU(),\n", + " nn.Linear(32, 1)\n", + " )\n", + "\n", + " def forward(self, x):\n", + " return self.layers(x)\n", + "\n", + " model = MLP().to(device)\n", + " model.load_state_dict(torch.load(model_dir, weights_only=True))\n", + "\n", + " # Preparing the inputs for dynamic batch sizing.\n", + " inputs = [trt.Input(min_shape=(20, 8), \n", + " opt_shape=(50, 8), \n", + " max_shape=(64, 8), \n", + " dtype=torch.float32)]\n", + "\n", + " # Trace the computation graph and compile to produce an optimized module\n", + " trt_gm = trt.compile(model, ir=\"dynamo\", inputs=inputs)\n", " \n", " def predict(inputs):\n", - " torch_inputs = torch.from_numpy(inputs).to(device)\n", - " outputs = scripted_mlp(torch_inputs) # .flatten()\n", - " return outputs.detach().numpy()\n", + " stream = torch.cuda.Stream()\n", + " with torch.no_grad(), torch.cuda.stream(stream), trt.logging.errors():\n", + " torch_inputs = torch.from_numpy(inputs).to(device)\n", + " outputs = trt_gm(torch_inputs) # .flatten()\n", + " return outputs.detach().cpu().numpy()\n", "\n", " return predict" ] }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 59, "id": "220a00a4-e842-4f5d-a4b3-7693d09e2d31", "metadata": {}, "outputs": [], @@ -1457,7 +1913,7 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 60, "id": "0f3bf287-8ffc-4456-8772-e97c418d6aee", "metadata": {}, "outputs": [ @@ -1472,8 +1928,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 200 ms, sys: 6.02 ms, total: 206 ms\n", - "Wall time: 3.24 s\n" + "CPU times: user 148 ms, sys: 17.9 ms, total: 166 ms\n", + "Wall time: 8.28 s\n" ] } ], @@ -1485,7 +1941,7 @@ }, { "cell_type": "code", - "execution_count": 53, + "execution_count": 61, "id": "6cd23b71-296d-4ce7-b56c-567cc2eec79c", "metadata": { "tags": [] @@ -1495,8 +1951,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 48.7 ms, sys: 2.68 ms, total: 51.4 ms\n", - "Wall time: 540 ms\n" + "CPU times: user 31.2 ms, sys: 5.84 ms, total: 37.1 ms\n", + "Wall time: 257 ms\n" ] } ], @@ -1508,7 +1964,7 @@ }, { "cell_type": "code", - "execution_count": 54, + "execution_count": 62, "id": "13c52980-fc55-4e81-ae54-b476b98f11b1", "metadata": {}, "outputs": [], @@ -1520,7 +1976,7 @@ }, { "cell_type": "code", - "execution_count": 55, + "execution_count": 63, "id": "764a40d8-25f7-425c-ba03-fe8c45f4b063", "metadata": {}, "outputs": [ @@ -1528,30 +1984,30 @@ "name": "stdout", "output_type": "stream", "text": [ - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+----------+\n", - "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude| preds|\n", - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+----------+\n", - "| -1.669445|-0.20972852| -1.1155425| -0.17418891| 0.2070965| -0.13437383| 0.5094524| -0.08001011| 0.8521974|\n", - "| -0.8564016| -1.5605159| -0.53141737| -0.02190494| -0.66006166| -0.12997007| 0.5141335| -0.08001011| 1.1964239|\n", - "| 0.73173314|-0.76593506| 0.67250663| -0.10979619| 0.14175056| -0.02296524| 0.5141335| -0.07002536| 1.7371099|\n", - "|-0.44887984| -1.719432| 0.14690235| -0.009905009| -0.06664997| 0.01090982|0.50476956|-0.075017735| 1.0491724|\n", - "|-0.96920437| -1.0837674| 0.058284093| 0.07351444| -0.20793848| -0.011342554| 0.5094524| -0.08001011| 0.8659589|\n", - "| -1.0906925| 0.26701993| -0.72011936| -0.2243873| -0.6574125| 0.021105729|0.49072453| -0.08001011|0.70745003|\n", - "|-0.90240705| 0.5848523| 0.24784403| 0.08091579| -0.5205393| 0.043980353| 0.4954056|-0.060044423| 0.5371985|\n", - "| -1.0712692|0.028645715| -0.38059494| 0.062626354| -0.5205393| -0.005065064|0.49072453| -0.09498342| 0.8064569|\n", - "|0.026544658| -0.686477| 0.06193017| -0.20868859| 0.03401808| 0.017589226| 0.4766795|-0.050059676| 1.1329681|\n", - "|-0.24211854| 0.34647804| 0.2122066| -0.11910806| -0.71127874| 0.047426607|0.47199664|-0.030093992|0.87097365|\n", - "| -1.1632799| -1.7988901| -0.3091| -0.16322336| -0.0719483| 0.01691941|0.46263272|-0.040078737| 0.6880104|\n", - "| -0.4509853| 0.8232265| -0.10236703| -0.053904843| -0.6953838| -0.03862959|0.46731555| -0.05505205| 0.9974648|\n", - "| -0.5102555|-0.76593506| 0.26753396| 0.023170268| -0.14877392|-0.0061120046| 0.4766795|-0.084998675| 1.0234879|\n", - "| -1.1635431| 0.18756187| -0.59673625| -0.023986263| -0.71481097| -0.014560648|0.46263272| -0.05505205|0.82353294|\n", - "|-0.15400293| 0.5053942| 0.33031902|-0.0012220128| -0.35275918| 0.0010038698|0.46263272| -0.08001011| 1.1798486|\n", - "|-0.27733323| 0.18756187|-0.010031368| -0.13782738| -0.32450148| 0.059288267| 0.4485877| -0.08001011| 0.8976056|\n", - "| -0.8956694| 0.18756187|-0.043201063| -0.12283955|-0.087843254| 0.011250444|0.49072453| -0.11494911|0.61913794|\n", - "|-0.57936895|-0.13027044| 0.15416715| 0.08602174| -0.6353362| 0.026508931| 0.4766795| -0.10496436| 0.8908371|\n", - "| -0.8548752| 0.1081038|-0.061331607| -0.26316243| -0.36865416| 0.0066948985|0.47199664| -0.12493005| 0.6539954|\n", - "| -1.1061679| 0.42593613| 0.053027753| -0.12104499| -0.2132368| -0.015556259| 0.4766795| -0.11494911|0.54294837|\n", - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+----------+\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+----------+\n", + "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude| preds|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+----------+\n", + "| 0.20909257| -1.1632254| 0.38946992| 0.04609274| -0.9806099| -0.07099328|0.61245227|-0.020113053| 1.3980268|\n", + "|-0.098627955| 0.34647804| 0.27216315| -0.0129226| -0.6953838| -0.05380849| 1.0665938| -1.2479742| 1.7447104|\n", + "| -0.66006273| 1.0616008|-0.55292207| -0.48945764|-0.13641118| 0.028952759| 1.1040496| -1.3827378| 1.439564|\n", + "| 0.08218294| 0.5848523|-0.13912922| -0.14707813|-0.19116047| -0.07136432|0.96827507| -1.3028787| 2.4199378|\n", + "| 0.0784456| -1.4810578| 0.57265776| 0.32067496| 1.0345173|-0.024157424| 1.4411427| -0.52423614| 1.2448893|\n", + "| -0.82318723| -0.36864465| 0.07829511| -0.1808107|-0.67242444|-0.061470542| 1.9374212| -1.0083897|0.68910843|\n", + "| 0.59671736| 0.5848523| 0.19346413| -0.1371872|-0.19645879| 0.009964322|0.96827507| -1.2928978| 2.656445|\n", + "| -0.9612035| -1.5605159|-0.56329846| 0.027148023|-0.71127874| -0.08471591| 0.5328614| -0.13990337| 1.13419|\n", + "| -0.74344087| -1.2426835| 0.27282518| 0.4037246| -0.9841421| -0.05610115| 1.2257773| -0.42940006| 1.1380601|\n", + "| 0.9784464| -0.2891866| 0.24374022| -0.24670053| 0.28922042| -0.01102468| 1.1087307| -1.2280084| 2.5711632|\n", + "| -0.5070446| -1.0043093|-0.78254056|0.0122275995| 2.8465424|-0.060435444| 0.8980464| -1.2080427| 1.8561494|\n", + "| -0.18690155| 1.2205169|0.015323491| 0.12183313|-0.41015765| 0.04452552| 1.010412| -1.3228445| 1.8643656|\n", + "| -1.2551856| 1.6178073| -0.3341509|-0.060125165| -0.7554314| -0.08777025| 1.0291398| -1.3477987| 1.2487215|\n", + "| 4.9607058| -1.9578062| 1.4854684| -0.03948475| 2.1833694|0.0029250523| 1.024457| -1.1581304| 5.595224|\n", + "| 0.73652315| -1.6399739| 0.7913185| -0.05238397| 1.67738| 0.01944797| 1.0993668| -1.1331724| 2.069084|\n", + "| -0.505834| 0.18756187|-0.47093546| -0.24297306|-0.60619545| -0.10791535| 0.977639| -1.2879055| 1.7858529|\n", + "| -0.88477343|-0.050812364| -0.6318951| -0.15244243| -0.5258376| -0.15618815| 0.9823201| -1.2879055| 1.6675146|\n", + "| -0.42840376| 0.9821427| -0.2266495| -0.36083496| -0.6883194| -0.08552282| 0.5328614| -0.12493005| 1.01702|\n", + "| 0.9369153| -1.4810578| 0.6722208|-0.121177554| 0.3996021| 0.01291408| 1.1040496| -1.1082181| 2.1314554|\n", + "| -0.80702734| -0.92485124|-0.26602685| -0.1560743| 1.4398388| -0.09314839|0.55627036| -0.09498342| 0.8631196|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+----------+\n", "only showing top 20 rows\n", "\n" ] @@ -1571,7 +2027,7 @@ }, { "cell_type": "code", - "execution_count": 56, + "execution_count": 64, "id": "1a69a9d2-5c7f-4e71-bb65-ae51927ccacf", "metadata": {}, "outputs": [], @@ -1583,7 +2039,7 @@ }, { "cell_type": "code", - "execution_count": 57, + "execution_count": 65, "id": "7214e2ac-fd2c-473e-a9c7-a65488570b5c", "metadata": {}, "outputs": [], @@ -1593,7 +2049,7 @@ }, { "cell_type": "code", - "execution_count": 58, + "execution_count": 66, "id": "5ee170b9-8ba6-4681-a10c-4cea71c1be15", "metadata": {}, "outputs": [ @@ -1610,7 +2066,7 @@ " 'Longitude']" ] }, - "execution_count": 58, + "execution_count": 66, "metadata": {}, "output_type": "execute_result" } @@ -1622,30 +2078,31 @@ }, { "cell_type": "code", - "execution_count": 59, + "execution_count": 67, "id": "0b7af043-5f20-49c7-bed6-39a9d13988e4", "metadata": {}, "outputs": [], "source": [ "# get absolute path to model\n", - "model2_dir = \"{}/housing_model2.ts\".format(os.getcwd())" + "model2_dir = \"{}/ts_housing_model2.pt\".format(os.getcwd())" ] }, { "cell_type": "code", - "execution_count": 60, + "execution_count": 68, "id": "104b2378-e191-4560-9a2e-276b8dcf0f2b", "metadata": {}, "outputs": [], "source": [ "def predict_batch_fn():\n", " import torch\n", + "\n", " device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n", - " print(\"Using {} device\".format(device))\n", " scripted_mlp = torch.jit.load(model2_dir)\n", " scripted_mlp.to(device)\n", " \n", " def predict(inc, age, rms, bdrms, pop, occ, lat, lon):\n", + " # print input shape\n", " outputs = scripted_mlp(\n", " torch.from_numpy(inc).to(device),\n", " torch.from_numpy(age).to(device),\n", @@ -1656,14 +2113,14 @@ " torch.from_numpy(lat).to(device),\n", " torch.from_numpy(lon).to(device),\n", " )\n", - " return outputs.detach().numpy()\n", + " return outputs.detach().cpu().numpy()\n", "\n", " return predict" ] }, { "cell_type": "code", - "execution_count": 61, + "execution_count": 69, "id": "020056dc-f8b0-483a-88eb-7e1ff2a0fdcf", "metadata": {}, "outputs": [], @@ -1675,7 +2132,7 @@ }, { "cell_type": "code", - "execution_count": 62, + "execution_count": 70, "id": "1b73518e-04ec-49c7-bf1e-93520d94028e", "metadata": {}, "outputs": [ @@ -1683,15 +2140,22 @@ "name": "stderr", "output_type": "stream", "text": [ - " \r" + "[Stage 12:==================================================> (7 + 1) / 8]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 192 ms, sys: 4.28 ms, total: 196 ms\n", - "Wall time: 1.89 s\n" + "CPU times: user 16.7 ms, sys: 5.44 ms, total: 22.1 ms\n", + "Wall time: 1.13 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" ] } ], @@ -1704,21 +2168,7 @@ }, { "cell_type": "code", - "execution_count": 63, - "id": "2d3559d3-644f-432e-92fc-21e6339a19f2", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "# should fail with ValueError\n", - "# preds = df.withColumn(\"preds\", classify(array(*columns)))\n", - "# results = preds.collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 64, + "execution_count": 71, "id": "86b56805-a211-43cb-878d-78957b08f865", "metadata": {}, "outputs": [ @@ -1726,8 +2176,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 64.7 ms, sys: 584 µs, total: 65.3 ms\n", - "Wall time: 363 ms\n" + "CPU times: user 19.6 ms, sys: 8.68 ms, total: 28.3 ms\n", + "Wall time: 451 ms\n" ] } ], @@ -1739,7 +2189,7 @@ }, { "cell_type": "code", - "execution_count": 65, + "execution_count": 72, "id": "5032b474-db92-4f04-b732-8b9d418cf211", "metadata": { "scrolled": true, @@ -1750,30 +2200,30 @@ "name": "stdout", "output_type": "stream", "text": [ - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+----------+\n", - "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude| preds|\n", - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+----------+\n", - "| -1.669445|-0.20972852| -1.1155425| -0.17418891| 0.2070965| -0.13437383| 0.5094524| -0.08001011| 0.8521974|\n", - "| -0.8564016| -1.5605159| -0.53141737| -0.02190494| -0.66006166| -0.12997007| 0.5141335| -0.08001011| 1.1964239|\n", - "| 0.73173314|-0.76593506| 0.67250663| -0.10979619| 0.14175056| -0.02296524| 0.5141335| -0.07002536| 1.7371099|\n", - "|-0.44887984| -1.719432| 0.14690235| -0.009905009| -0.06664997| 0.01090982|0.50476956|-0.075017735| 1.0491724|\n", - "|-0.96920437| -1.0837674| 0.058284093| 0.07351444| -0.20793848| -0.011342554| 0.5094524| -0.08001011| 0.8659589|\n", - "| -1.0906925| 0.26701993| -0.72011936| -0.2243873| -0.6574125| 0.021105729|0.49072453| -0.08001011|0.70745003|\n", - "|-0.90240705| 0.5848523| 0.24784403| 0.08091579| -0.5205393| 0.043980353| 0.4954056|-0.060044423| 0.5371985|\n", - "| -1.0712692|0.028645715| -0.38059494| 0.062626354| -0.5205393| -0.005065064|0.49072453| -0.09498342| 0.8064569|\n", - "|0.026544658| -0.686477| 0.06193017| -0.20868859| 0.03401808| 0.017589226| 0.4766795|-0.050059676| 1.1329681|\n", - "|-0.24211854| 0.34647804| 0.2122066| -0.11910806| -0.71127874| 0.047426607|0.47199664|-0.030093992|0.87097365|\n", - "| -1.1632799| -1.7988901| -0.3091| -0.16322336| -0.0719483| 0.01691941|0.46263272|-0.040078737| 0.6880104|\n", - "| -0.4509853| 0.8232265| -0.10236703| -0.053904843| -0.6953838| -0.03862959|0.46731555| -0.05505205| 0.9974648|\n", - "| -0.5102555|-0.76593506| 0.26753396| 0.023170268| -0.14877392|-0.0061120046| 0.4766795|-0.084998675| 1.0234879|\n", - "| -1.1635431| 0.18756187| -0.59673625| -0.023986263| -0.71481097| -0.014560648|0.46263272| -0.05505205|0.82353294|\n", - "|-0.15400293| 0.5053942| 0.33031902|-0.0012220128| -0.35275918| 0.0010038698|0.46263272| -0.08001011| 1.1798486|\n", - "|-0.27733323| 0.18756187|-0.010031368| -0.13782738| -0.32450148| 0.059288267| 0.4485877| -0.08001011| 0.8976056|\n", - "| -0.8956694| 0.18756187|-0.043201063| -0.12283955|-0.087843254| 0.011250444|0.49072453| -0.11494911|0.61913794|\n", - "|-0.57936895|-0.13027044| 0.15416715| 0.08602174| -0.6353362| 0.026508931| 0.4766795| -0.10496436| 0.8908371|\n", - "| -0.8548752| 0.1081038|-0.061331607| -0.26316243| -0.36865416| 0.0066948985|0.47199664| -0.12493005| 0.6539954|\n", - "| -1.1061679| 0.42593613| 0.053027753| -0.12104499| -0.2132368| -0.015556259| 0.4766795| -0.11494911|0.54294837|\n", - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+----------+\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+----------+\n", + "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude| preds|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+----------+\n", + "| 0.20909257| -1.1632254| 0.38946992| 0.04609274| -0.9806099| -0.07099328|0.61245227|-0.020113053| 1.3979516|\n", + "|-0.098627955| 0.34647804| 0.27216315| -0.0129226| -0.6953838| -0.05380849| 1.0665938| -1.2479742| 1.7442212|\n", + "| -0.66006273| 1.0616008|-0.55292207| -0.48945764|-0.13641118| 0.028952759| 1.1040496| -1.3827378| 1.4398992|\n", + "| 0.08218294| 0.5848523|-0.13912922| -0.14707813|-0.19116047| -0.07136432|0.96827507| -1.3028787| 2.4199052|\n", + "| 0.0784456| -1.4810578| 0.57265776| 0.32067496| 1.0345173|-0.024157424| 1.4411427| -0.52423614| 1.2446644|\n", + "| -0.82318723| -0.36864465| 0.07829511| -0.1808107|-0.67242444|-0.061470542| 1.9374212| -1.0083897| 0.6888372|\n", + "| 0.59671736| 0.5848523| 0.19346413| -0.1371872|-0.19645879| 0.009964322|0.96827507| -1.2928978| 2.6563153|\n", + "| -0.9612035| -1.5605159|-0.56329846| 0.027148023|-0.71127874| -0.08471591| 0.5328614| -0.13990337| 1.1341839|\n", + "| -0.74344087| -1.2426835| 0.27282518| 0.4037246| -0.9841421| -0.05610115| 1.2257773| -0.42940006| 1.1378745|\n", + "| 0.9784464| -0.2891866| 0.24374022| -0.24670053| 0.28922042| -0.01102468| 1.1087307| -1.2280084| 2.5710382|\n", + "| -0.5070446| -1.0043093|-0.78254056|0.0122275995| 2.8465424|-0.060435444| 0.8980464| -1.2080427| 1.8561647|\n", + "| -0.18690155| 1.2205169|0.015323491| 0.12183313|-0.41015765| 0.04452552| 1.010412| -1.3228445| 1.8639375|\n", + "| -1.2551856| 1.6178073| -0.3341509|-0.060125165| -0.7554314| -0.08777025| 1.0291398| -1.3477987| 1.24879|\n", + "| 4.9607058| -1.9578062| 1.4854684| -0.03948475| 2.1833694|0.0029250523| 1.024457| -1.1581304| 5.5946765|\n", + "| 0.73652315| -1.6399739| 0.7913185| -0.05238397| 1.67738| 0.01944797| 1.0993668| -1.1331724| 2.0694952|\n", + "| -0.505834| 0.18756187|-0.47093546| -0.24297306|-0.60619545| -0.10791535| 0.977639| -1.2879055| 1.7852836|\n", + "| -0.88477343|-0.050812364| -0.6318951| -0.15244243| -0.5258376| -0.15618815| 0.9823201| -1.2879055| 1.6675682|\n", + "| -0.42840376| 0.9821427| -0.2266495| -0.36083496| -0.6883194| -0.08552282| 0.5328614| -0.12493005| 1.0169572|\n", + "| 0.9369153| -1.4810578| 0.6722208|-0.121177554| 0.3996021| 0.01291408| 1.1040496| -1.1082181| 2.1317377|\n", + "| -0.80702734| -0.92485124|-0.26602685| -0.1560743| 1.4398388| -0.09314839|0.55627036| -0.09498342|0.86303747|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+----------+\n", "only showing top 20 rows\n", "\n" ] @@ -1795,7 +2245,7 @@ }, { "cell_type": "code", - "execution_count": 66, + "execution_count": 73, "id": "a9ab4cdf-8103-447e-9ac8-944e2e527239", "metadata": {}, "outputs": [], @@ -1810,7 +2260,7 @@ }, { "cell_type": "code", - "execution_count": 67, + "execution_count": 74, "id": "6632636e-67a3-406c-832c-758aac4245fd", "metadata": {}, "outputs": [], @@ -1821,7 +2271,7 @@ "mkdir models\n", "cp -r models_config/housing_model models\n", "mkdir -p models/housing_model/1\n", - "cp housing_model.ts models/housing_model/1/model.pt" + "cp ts_housing_model.pt models/housing_model/1/model.pt" ] }, { @@ -1834,7 +2284,7 @@ }, { "cell_type": "code", - "execution_count": 68, + "execution_count": 75, "id": "c6fd1612-de6a-461c-a2ad-1a3fcd277d66", "metadata": { "scrolled": true @@ -1853,7 +2303,7 @@ "[True]" ] }, - "execution_count": 68, + "execution_count": 75, "metadata": {}, "output_type": "execute_result" } @@ -1874,7 +2324,7 @@ " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", " else:\n", " container=client.containers.run(\n", - " \"nvcr.io/nvidia/tritonserver:22.07-py3\", \"tritonserver --model-repository=/models\",\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", " detach=True,\n", " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", " name=\"spark-triton\",\n", @@ -1910,7 +2360,7 @@ }, { "cell_type": "code", - "execution_count": 69, + "execution_count": 76, "id": "5eae04bc-75ca-421a-87c8-ac507ce1f2f5", "metadata": {}, "outputs": [], @@ -1920,7 +2370,7 @@ }, { "cell_type": "code", - "execution_count": 70, + "execution_count": 77, "id": "b350bd8e-9b8f-4511-9ddf-76d917b21b5f", "metadata": { "tags": [] @@ -1939,7 +2389,7 @@ " 'Longitude']" ] }, - "execution_count": 70, + "execution_count": 77, "metadata": {}, "output_type": "execute_result" } @@ -1951,7 +2401,7 @@ }, { "cell_type": "code", - "execution_count": 71, + "execution_count": 78, "id": "69b343ec-688d-4e4d-985e-db72beaaf00c", "metadata": {}, "outputs": [], @@ -1961,7 +2411,7 @@ " import tritonclient.grpc as grpcclient\n", " \n", " np_types = {\n", - " \"BOOL\": np.dtype(np.bool8),\n", + " \"BOOL\": np.dtype(np.bool_),\n", " \"INT8\": np.dtype(np.int8),\n", " \"INT16\": np.dtype(np.int16),\n", " \"INT32\": np.dtype(np.int32),\n", @@ -2001,7 +2451,7 @@ }, { "cell_type": "code", - "execution_count": 72, + "execution_count": 79, "id": "d3e64fda-117b-4810-a9a2-dd498239496f", "metadata": {}, "outputs": [], @@ -2014,30 +2464,16 @@ }, { "cell_type": "code", - "execution_count": 73, + "execution_count": 80, "id": "a24149a5-3adc-4089-8769-13cf1e44547a", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[Stage 13:> (0 + 8) / 8]\r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 74.3 ms, sys: 9.28 ms, total: 83.6 ms\n", - "Wall time: 1.07 s\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" + "CPU times: user 15.3 ms, sys: 4.33 ms, total: 19.6 ms\n", + "Wall time: 461 ms\n" ] } ], @@ -2050,7 +2486,7 @@ }, { "cell_type": "code", - "execution_count": 74, + "execution_count": 81, "id": "df2ce39f-30af-491a-8472-800fb1ce8458", "metadata": {}, "outputs": [ @@ -2058,8 +2494,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 197 ms, sys: 24.2 ms, total: 221 ms\n", - "Wall time: 400 ms\n" + "CPU times: user 30.2 ms, sys: 5.78 ms, total: 36 ms\n", + "Wall time: 200 ms\n" ] } ], @@ -2071,7 +2507,7 @@ }, { "cell_type": "code", - "execution_count": 75, + "execution_count": 82, "id": "ca6f3eaa-9569-45d0-88bf-9aa0757e1ecb", "metadata": {}, "outputs": [], @@ -2083,7 +2519,7 @@ }, { "cell_type": "code", - "execution_count": 76, + "execution_count": 83, "id": "b79c62c8-e1e8-4467-8aef-8939c31833b8", "metadata": { "tags": [] @@ -2093,30 +2529,30 @@ "name": "stdout", "output_type": "stream", "text": [ - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+----------+\n", - "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude| preds|\n", - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+----------+\n", - "| -1.669445|-0.20972852| -1.1155425| -0.17418891| 0.2070965| -0.13437383| 0.5094524| -0.08001011|0.85219747|\n", - "| -0.8564016| -1.5605159| -0.53141737| -0.02190494| -0.66006166| -0.12997007| 0.5141335| -0.08001011| 1.1964238|\n", - "| 0.73173314|-0.76593506| 0.67250663| -0.10979619| 0.14175056| -0.02296524| 0.5141335| -0.07002536| 1.7371097|\n", - "|-0.44887984| -1.719432| 0.14690235| -0.009905009| -0.06664997| 0.01090982|0.50476956|-0.075017735| 1.0491724|\n", - "|-0.96920437| -1.0837674| 0.058284093| 0.07351444| -0.20793848| -0.011342554| 0.5094524| -0.08001011| 0.8659591|\n", - "| -1.0906925| 0.26701993| -0.72011936| -0.2243873| -0.6574125| 0.021105729|0.49072453| -0.08001011| 0.7074499|\n", - "|-0.90240705| 0.5848523| 0.24784403| 0.08091579| -0.5205393| 0.043980353| 0.4954056|-0.060044423|0.53719854|\n", - "| -1.0712692|0.028645715| -0.38059494| 0.062626354| -0.5205393| -0.005065064|0.49072453| -0.09498342|0.80645704|\n", - "|0.026544658| -0.686477| 0.06193017| -0.20868859| 0.03401808| 0.017589226| 0.4766795|-0.050059676| 1.1329681|\n", - "|-0.24211854| 0.34647804| 0.2122066| -0.11910806| -0.71127874| 0.047426607|0.47199664|-0.030093992| 0.8709737|\n", - "| -1.1632799| -1.7988901| -0.3091| -0.16322336| -0.0719483| 0.01691941|0.46263272|-0.040078737| 0.6880103|\n", - "| -0.4509853| 0.8232265| -0.10236703| -0.053904843| -0.6953838| -0.03862959|0.46731555| -0.05505205| 0.9974648|\n", - "| -0.5102555|-0.76593506| 0.26753396| 0.023170268| -0.14877392|-0.0061120046| 0.4766795|-0.084998675| 1.0234879|\n", - "| -1.1635431| 0.18756187| -0.59673625| -0.023986263| -0.71481097| -0.014560648|0.46263272| -0.05505205| 0.823533|\n", - "|-0.15400293| 0.5053942| 0.33031902|-0.0012220128| -0.35275918| 0.0010038698|0.46263272| -0.08001011| 1.1798486|\n", - "|-0.27733323| 0.18756187|-0.010031368| -0.13782738| -0.32450148| 0.059288267| 0.4485877| -0.08001011| 0.8976056|\n", - "| -0.8956694| 0.18756187|-0.043201063| -0.12283955|-0.087843254| 0.011250444|0.49072453| -0.11494911|0.61913794|\n", - "|-0.57936895|-0.13027044| 0.15416715| 0.08602174| -0.6353362| 0.026508931| 0.4766795| -0.10496436| 0.890837|\n", - "| -0.8548752| 0.1081038|-0.061331607| -0.26316243| -0.36865416| 0.0066948985|0.47199664| -0.12493005| 0.6539954|\n", - "| -1.1061679| 0.42593613| 0.053027753| -0.12104499| -0.2132368| -0.015556259| 0.4766795| -0.11494911| 0.5429483|\n", - "+-----------+-----------+------------+-------------+------------+-------------+----------+------------+----------+\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+----------+\n", + "| MedInc| HouseAge| AveRooms| AveBedrms| Population| AveOccup| Latitude| Longitude| preds|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+----------+\n", + "| 0.20909257| -1.1632254| 0.38946992| 0.04609274| -0.9806099| -0.07099328|0.61245227|-0.020113053| 1.3979516|\n", + "|-0.098627955| 0.34647804| 0.27216315| -0.0129226| -0.6953838| -0.05380849| 1.0665938| -1.2479742| 1.7442212|\n", + "| -0.66006273| 1.0616008|-0.55292207| -0.48945764|-0.13641118| 0.028952759| 1.1040496| -1.3827378| 1.4398992|\n", + "| 0.08218294| 0.5848523|-0.13912922| -0.14707813|-0.19116047| -0.07136432|0.96827507| -1.3028787| 2.4199052|\n", + "| 0.0784456| -1.4810578| 0.57265776| 0.32067496| 1.0345173|-0.024157424| 1.4411427| -0.52423614| 1.2446644|\n", + "| -0.82318723| -0.36864465| 0.07829511| -0.1808107|-0.67242444|-0.061470542| 1.9374212| -1.0083897| 0.6888372|\n", + "| 0.59671736| 0.5848523| 0.19346413| -0.1371872|-0.19645879| 0.009964322|0.96827507| -1.2928978| 2.6563153|\n", + "| -0.9612035| -1.5605159|-0.56329846| 0.027148023|-0.71127874| -0.08471591| 0.5328614| -0.13990337| 1.1341839|\n", + "| -0.74344087| -1.2426835| 0.27282518| 0.4037246| -0.9841421| -0.05610115| 1.2257773| -0.42940006| 1.1378745|\n", + "| 0.9784464| -0.2891866| 0.24374022| -0.24670053| 0.28922042| -0.01102468| 1.1087307| -1.2280084| 2.5710382|\n", + "| -0.5070446| -1.0043093|-0.78254056|0.0122275995| 2.8465424|-0.060435444| 0.8980464| -1.2080427| 1.8561647|\n", + "| -0.18690155| 1.2205169|0.015323491| 0.12183313|-0.41015765| 0.04452552| 1.010412| -1.3228445| 1.8639375|\n", + "| -1.2551856| 1.6178073| -0.3341509|-0.060125165| -0.7554314| -0.08777025| 1.0291398| -1.3477987| 1.24879|\n", + "| 4.9607058| -1.9578062| 1.4854684| -0.03948475| 2.1833694|0.0029250523| 1.024457| -1.1581304| 5.5946765|\n", + "| 0.73652315| -1.6399739| 0.7913185| -0.05238397| 1.67738| 0.01944797| 1.0993668| -1.1331724| 2.0694952|\n", + "| -0.505834| 0.18756187|-0.47093546| -0.24297306|-0.60619545| -0.10791535| 0.977639| -1.2879055| 1.7852836|\n", + "| -0.88477343|-0.050812364| -0.6318951| -0.15244243| -0.5258376| -0.15618815| 0.9823201| -1.2879055| 1.6675682|\n", + "| -0.42840376| 0.9821427| -0.2266495| -0.36083496| -0.6883194| -0.08552282| 0.5328614| -0.12493005| 1.0169572|\n", + "| 0.9369153| -1.4810578| 0.6722208|-0.121177554| 0.3996021| 0.01291408| 1.1040496| -1.1082181| 2.1317377|\n", + "| -0.80702734| -0.92485124|-0.26602685| -0.1560743| 1.4398388| -0.09314839|0.55627036| -0.09498342|0.86303747|\n", + "+------------+------------+-----------+------------+-----------+------------+----------+------------+----------+\n", "only showing top 20 rows\n", "\n" ] @@ -2138,7 +2574,7 @@ }, { "cell_type": "code", - "execution_count": 77, + "execution_count": 84, "id": "15e9b3df-f3c9-46bb-bbeb-42496f7663de", "metadata": {}, "outputs": [ @@ -2155,7 +2591,7 @@ "[True]" ] }, - "execution_count": 77, + "execution_count": 84, "metadata": {}, "output_type": "execute_result" } @@ -2179,7 +2615,7 @@ }, { "cell_type": "code", - "execution_count": 78, + "execution_count": 85, "id": "0138a029-87c5-497f-ac5c-3eed0e11b0f6", "metadata": {}, "outputs": [], @@ -2198,7 +2634,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "spark-dl-torch", "language": "python", "name": "python3" }, @@ -2212,7 +2648,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.10" + "version": "3.11.9" } }, "nbformat": 4, diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/requirements.txt b/examples/ML+DL-Examples/Spark-DL/dl_inference/requirements.txt index 87c8dcfbe..abd6ad089 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/requirements.txt +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/requirements.txt @@ -1,19 +1,16 @@ -docker -huggingface -jupyterlab -matplotlib numpy pandas +matplotlib portalocker pyarrow -pyspark>=3.4.0 -sentencepiece -sentence_transformers +h5py +pydot scikit-learn -tensorflow-cpu +jupyterlab +pyspark>=3.4.0 +huggingface +datasets +docker +tritonclient[grpc] transformers -tritonclient ---extra-index-url https://download.pytorch.org/whl/cpu -torch -torchtext -torchvision +ipywidgets \ No newline at end of file diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/feature_columns.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/feature_columns_tf.ipynb similarity index 63% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/feature_columns.ipynb rename to examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/feature_columns_tf.ipynb index 2a338d4d8..17af1c934 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/feature_columns.ipynb +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/feature_columns_tf.ipynb @@ -2,18 +2,45 @@ "cells": [ { "cell_type": "markdown", - "id": "66834048-5285-44a9-a031-e2e6e3347442", + "id": "7fcc021a", "metadata": {}, "source": [ + "# Pyspark TensorFlow Inference\n", + "\n", + "## Feature Columns\n", "From: https://www.tensorflow.org/tutorials/structured_data/preprocessing_layers" ] }, + { + "cell_type": "markdown", + "id": "35203476", + "metadata": {}, + "source": [ + "### Using TensorFlow\n", + "Note that cuFFT/cuDNN/cuBLAS registration errors are expected with `tf=2.17.0` and will not affect behavior, as noted in [this issue.](https://github.com/tensorflow/tensorflow/issues/62075) \n", + "This notebook does not demonstrate inference with TensorRT, as [TF-TRT](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#tensorrt-10) does not yet support `tf=2.17.0`. See the `pytorch` notebooks for TensorRT demos." + ] + }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "id": "01162f42-0637-4dfe-8d7d-b577e4ffd017", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:38:52.548855: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", + "2024-10-03 17:38:52.555529: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", + "2024-10-03 17:38:52.563119: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", + "2024-10-03 17:38:52.565499: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n", + "2024-10-03 17:38:52.571252: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", + "To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", + "2024-10-03 17:38:52.894224: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" + ] + } + ], "source": [ "import numpy as np\n", "import pandas as pd\n", @@ -24,28 +51,34 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 3, "id": "9fa3e1b7-58cd-45f9-9fee-85f25a31c3c6", "metadata": {}, "outputs": [ { - "data": { - "text/plain": [ - "'2.12.0'" - ] - }, - "execution_count": 2, - "metadata": {}, - "output_type": "execute_result" + "name": "stdout", + "output_type": "stream", + "text": [ + "2.17.0\n" + ] } ], "source": [ - "tf.__version__" + "print(tf.__version__)\n", + "\n", + "# Enable GPU memory growth\n", + "gpus = tf.config.experimental.list_physical_devices('GPU')\n", + "if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)" ] }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 4, "id": "9326b072-a53c-40c4-a6cb-bd4d3d644d03", "metadata": {}, "outputs": [], @@ -60,7 +93,7 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 5, "id": "e98480ef-d13d-44c0-a227-e9a22f9bf2b0", "metadata": {}, "outputs": [ @@ -198,15 +231,15 @@ "" ], "text/plain": [ - " Type Age Breed1 Gender Color1 Color2 MaturitySize \n", - "0 Cat 3 Tabby Male Black White Small \\\n", + " Type Age Breed1 Gender Color1 Color2 MaturitySize \\\n", + "0 Cat 3 Tabby Male Black White Small \n", "1 Cat 1 Domestic Medium Hair Male Black Brown Medium \n", "2 Dog 1 Mixed Breed Male Brown White Medium \n", "3 Dog 4 Mixed Breed Female Black Brown Medium \n", "4 Dog 1 Mixed Breed Male Black No Color Medium \n", "\n", - " FurLength Vaccinated Sterilized Health Fee \n", - "0 Short No No Healthy 100 \\\n", + " FurLength Vaccinated Sterilized Health Fee \\\n", + "0 Short No No Healthy 100 \n", "1 Medium Not Sure Not Sure Healthy 0 \n", "2 Medium Yes No Healthy 0 \n", "3 Short Yes No Healthy 150 \n", @@ -220,7 +253,7 @@ "4 This handsome yet cute boy is up for adoption.... 3 2 " ] }, - "execution_count": 4, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } @@ -231,7 +264,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 6, "id": "e8efce25-a835-4cbd-b8a2-1418ba2c1d31", "metadata": {}, "outputs": [], @@ -246,17 +279,26 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 7, "id": "00d403cf-9ae7-4780-9fac-13d920d8b395", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/rishic/anaconda3/envs/spark-dl-tf/lib/python3.11/site-packages/numpy/core/fromnumeric.py:59: FutureWarning: 'DataFrame.swapaxes' is deprecated and will be removed in a future version. Please use 'DataFrame.transpose' instead.\n", + " return bound(*args, **kwds)\n" + ] + } + ], "source": [ - "train, val, test = np.split(dataframe.sample(frac=1), [int(0.8*len(dataframe)), int(0.9*len(dataframe))])\n" + "train, val, test = np.split(dataframe.sample(frac=1), [int(0.8*len(dataframe)), int(0.9*len(dataframe))])" ] }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 8, "id": "4206a56e-5403-42a9-805e-e037044e7995", "metadata": {}, "outputs": [ @@ -278,7 +320,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 9, "id": "499ade5f-ac8a-47ca-a021-071239dfe97d", "metadata": {}, "outputs": [], @@ -286,7 +328,7 @@ "def df_to_dataset(dataframe, shuffle=True, batch_size=32):\n", " df = dataframe.copy()\n", " labels = df.pop('target')\n", - " df = {key: np.array(value)[:, tf.newaxis] for key, value in dataframe.items()}\n", + " df = {key: value.to_numpy()[:,tf.newaxis] for key, value in dataframe.items()}\n", " ds = tf.data.Dataset.from_tensor_slices((dict(df), labels))\n", " if shuffle:\n", " ds = ds.shuffle(buffer_size=len(dataframe))\n", @@ -297,10 +339,18 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 10, "id": "b9ec57c9-080e-4626-9e03-acf309cf3736", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:38:53.526119: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2021] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 46022 MB memory: -> device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:01:00.0, compute capability: 8.6\n" + ] + } + ], "source": [ "batch_size = 5\n", "train_ds = df_to_dataset(train, batch_size=batch_size)" @@ -308,7 +358,7 @@ }, { "cell_type": "code", - "execution_count": 10, + "execution_count": 11, "id": "dfcbf268-4508-4eb8-abe1-acf1dbb97bd5", "metadata": {}, "outputs": [ @@ -318,12 +368,19 @@ "text": [ "Every feature: ['Type', 'Age', 'Breed1', 'Gender', 'Color1', 'Color2', 'MaturitySize', 'FurLength', 'Vaccinated', 'Sterilized', 'Health', 'Fee', 'PhotoAmt', 'target']\n", "A batch of ages: tf.Tensor(\n", - "[[ 2]\n", - " [15]\n", - " [ 3]\n", - " [11]\n", - " [ 8]], shape=(5, 1), dtype=int64)\n", - "A batch of targets: tf.Tensor([1 1 1 1 1], shape=(5,), dtype=int64)\n" + "[[18]\n", + " [ 5]\n", + " [ 2]\n", + " [ 5]\n", + " [ 1]], shape=(5, 1), dtype=int64)\n", + "A batch of targets: tf.Tensor([1 0 1 1 1], shape=(5,), dtype=int64)\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:38:53.588272: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence\n" ] } ], @@ -336,7 +393,7 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 12, "id": "6c09dc4b-3a2a-44f5-b41c-821ec30b87b1", "metadata": {}, "outputs": [], @@ -356,22 +413,29 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 13, "id": "59bb91dc-360a-4a89-a9ea-bebc1ddbf1b7", "metadata": {}, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:38:55.015073: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence\n" + ] + }, { "data": { "text/plain": [ "" + "array([[-0.19597158],\n", + " [-0.19597158],\n", + " [-0.8320273 ],\n", + " [-0.19597158],\n", + " [-0.8320273 ]], dtype=float32)>" ] }, - "execution_count": 12, + "execution_count": 13, "metadata": {}, "output_type": "execute_result" } @@ -384,7 +448,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 14, "id": "4623b612-e924-472b-9ef4-c7f14f9f53c5", "metadata": {}, "outputs": [], @@ -413,7 +477,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 15, "id": "0a40e9ee-20a5-4a42-8543-c267f99af55e", "metadata": {}, "outputs": [ @@ -421,14 +485,14 @@ "data": { "text/plain": [ "" ] }, - "execution_count": 14, + "execution_count": 15, "metadata": {}, "output_type": "execute_result" } @@ -443,22 +507,29 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 16, "id": "ff63a5cc-71f4-428e-9299-a8018edc7648", "metadata": {}, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:38:56.454126: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence\n" + ] + }, { "data": { "text/plain": [ "" + " [0., 0., 0., 1., 0.]], dtype=float32)>" ] }, - "execution_count": 15, + "execution_count": 16, "metadata": {}, "output_type": "execute_result" } @@ -474,7 +545,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 17, "id": "2b040b0e-d8ca-4cf0-917c-dd9a272e1f0a", "metadata": {}, "outputs": [], @@ -487,12 +558,12 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 18, "id": "19df498e-4dd1-467a-8741-e1f5e15932a5", "metadata": {}, "outputs": [], "source": [ - "all_inputs = []\n", + "all_inputs = {}\n", "encoded_features = []\n", "\n", "# Numerical features.\n", @@ -500,13 +571,13 @@ " numeric_col = tf.keras.Input(shape=(1,), name=header)\n", " normalization_layer = get_normalization_layer(header, train_ds)\n", " encoded_numeric_col = normalization_layer(numeric_col)\n", - " all_inputs.append(numeric_col)\n", + " all_inputs[header] = numeric_col\n", " encoded_features.append(encoded_numeric_col)" ] }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 19, "id": "1d12579f-34fb-40b0-a16a-3e13cfea8178", "metadata": {}, "outputs": [], @@ -518,16 +589,25 @@ " dtype='int64',\n", " max_tokens=5)\n", "encoded_age_col = encoding_layer(age_col)\n", - "all_inputs.append(age_col)\n", + "all_inputs['Age'] = age_col\n", "encoded_features.append(encoded_age_col)" ] }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 20, "id": "bff286eb-7ad7-4d3a-8fa4-c729692d1425", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:38:56.758056: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence\n", + "2024-10-03 17:38:57.171981: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence\n" + ] + } + ], "source": [ "categorical_cols = ['Type', 'Color1', 'Color2', 'Gender', 'MaturitySize',\n", " 'FurLength', 'Vaccinated', 'Sterilized', 'Health', 'Breed1']\n", @@ -535,18 +615,17 @@ "for header in categorical_cols:\n", " categorical_col = tf.keras.Input(shape=(1,), name=header, dtype='string')\n", " encoding_layer = get_category_encoding_layer(name=header,\n", - " dataset=train_ds,\n", - " dtype='string',\n", - " max_tokens=5)\n", - "\n", + " dataset=train_ds,\n", + " dtype='string',\n", + " max_tokens=5)\n", " encoded_categorical_col = encoding_layer(categorical_col)\n", - " all_inputs.append(categorical_col)\n", + " all_inputs[header] = categorical_col\n", " encoded_features.append(encoded_categorical_col)" ] }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 21, "id": "79247436-32d8-4738-a656-3f288c77001c", "metadata": {}, "outputs": [], @@ -561,33 +640,15 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 22, "id": "dbc85d3e-6d1e-4167-9516-b1182e880542", "metadata": {}, "outputs": [], "source": [ "model.compile(optimizer='adam',\n", " loss=tf.keras.losses.BinaryCrossentropy(from_logits=True),\n", - " metrics=[\"accuracy\"])" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "id": "03b54d91-26f3-4232-ac33-895599bd6126", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "You must install pydot (`pip install pydot`) and install graphviz (see instructions at https://graphviz.gitlab.io/download/) for plot_model to work.\n" - ] - } - ], - "source": [ - "# Use `rankdir='LR'` to make the graph horizontal.\n", - "tf.keras.utils.plot_model(model, show_shapes=True, rankdir=\"LR\")" + " metrics=[\"accuracy\"],\n", + " run_eagerly=True)" ] }, { @@ -603,43 +664,35 @@ "Epoch 1/10\n" ] }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "/home/leey/.pyenv/versions/3.9.10/envs/spark_rapids_examples/lib/python3.9/site-packages/keras/engine/functional.py:639: UserWarning: Input dict contained keys ['target'] which did not match any model input. They will be ignored by the model.\n", - " inputs = self._flatten_to_reference_inputs(inputs)\n" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "37/37 [==============================] - 1s 14ms/step - loss: 0.6070 - accuracy: 0.6274 - val_loss: 0.5371 - val_accuracy: 0.7374\n", + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 15ms/step - accuracy: 0.4109 - loss: 0.7333 - val_accuracy: 0.6898 - val_loss: 0.5666\n", "Epoch 2/10\n", - "37/37 [==============================] - 0s 4ms/step - loss: 0.5733 - accuracy: 0.6728 - val_loss: 0.5245 - val_accuracy: 0.7366\n", + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 15ms/step - accuracy: 0.6423 - loss: 0.5994 - val_accuracy: 0.7210 - val_loss: 0.5484\n", "Epoch 3/10\n", - "37/37 [==============================] - 0s 5ms/step - loss: 0.5552 - accuracy: 0.6964 - val_loss: 0.5158 - val_accuracy: 0.7409\n", + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 17ms/step - accuracy: 0.6825 - loss: 0.5728 - val_accuracy: 0.7253 - val_loss: 0.5383\n", "Epoch 4/10\n", - "37/37 [==============================] - 0s 6ms/step - loss: 0.5469 - accuracy: 0.7025 - val_loss: 0.5116 - val_accuracy: 0.7470\n", + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 17ms/step - accuracy: 0.6796 - loss: 0.5653 - val_accuracy: 0.7331 - val_loss: 0.5314\n", "Epoch 5/10\n", - "37/37 [==============================] - 0s 6ms/step - loss: 0.5419 - accuracy: 0.7097 - val_loss: 0.5090 - val_accuracy: 0.7409\n", + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 16ms/step - accuracy: 0.6853 - loss: 0.5584 - val_accuracy: 0.7348 - val_loss: 0.5259\n", "Epoch 6/10\n", - "37/37 [==============================] - 0s 5ms/step - loss: 0.5356 - accuracy: 0.7142 - val_loss: 0.5089 - val_accuracy: 0.7461\n", + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 14ms/step - accuracy: 0.7120 - loss: 0.5447 - val_accuracy: 0.7418 - val_loss: 0.5218\n", "Epoch 7/10\n", - "37/37 [==============================] - 0s 6ms/step - loss: 0.5327 - accuracy: 0.7073 - val_loss: 0.5056 - val_accuracy: 0.7530\n", + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 14ms/step - accuracy: 0.7068 - loss: 0.5422 - val_accuracy: 0.7435 - val_loss: 0.5189\n", "Epoch 8/10\n", - "37/37 [==============================] - 0s 5ms/step - loss: 0.5319 - accuracy: 0.7148 - val_loss: 0.5039 - val_accuracy: 0.7574\n", + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 14ms/step - accuracy: 0.7043 - loss: 0.5397 - val_accuracy: 0.7435 - val_loss: 0.5162\n", "Epoch 9/10\n", - "37/37 [==============================] - 0s 6ms/step - loss: 0.5229 - accuracy: 0.7223 - val_loss: 0.5042 - val_accuracy: 0.7565\n", + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 14ms/step - accuracy: 0.7172 - loss: 0.5372 - val_accuracy: 0.7496 - val_loss: 0.5146\n", "Epoch 10/10\n", - "37/37 [==============================] - 0s 6ms/step - loss: 0.5242 - accuracy: 0.7200 - val_loss: 0.5023 - val_accuracy: 0.7582\n" + "\u001b[1m37/37\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 14ms/step - accuracy: 0.7337 - loss: 0.5232 - val_accuracy: 0.7409 - val_loss: 0.5131\n" ] }, { "data": { "text/plain": [ - "" + "" ] }, "execution_count": 23, @@ -661,8 +714,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "5/5 [==============================] - 0s 5ms/step - loss: 0.4920 - accuracy: 0.7504\n", - "Accuracy 0.7504332661628723\n" + "\u001b[1m5/5\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 7ms/step - accuracy: 0.7480 - loss: 0.5028 \n", + "Accuracy 0.753032922744751\n" ] } ], @@ -692,82 +745,26 @@ { "cell_type": "code", "execution_count": 26, - "id": "d1724a4e-fdaa-4169-8740-05bf18cb3153", + "id": "6bf0d024", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING:absl:Function `_wrapped_model` contains input name(s) PhotoAmt, Fee, Age, Type, Color1, Color2, Gender, MaturitySize, FurLength, Vaccinated, Sterilized, Health, Breed1 with unsupported characters which will be renamed to photoamt, fee, age, type, color1, color2, gender, maturitysize, furlength, vaccinated, sterilized, health, breed1 in the SavedModel.\n", - "WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "INFO:tensorflow:Assets written to: my_pet_classifier/assets\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "INFO:tensorflow:Assets written to: my_pet_classifier/assets\n" - ] - } - ], + "outputs": [], "source": [ - "model.save('my_pet_classifier')" + "model.save('my_pet_classifier.keras')" ] }, { "cell_type": "code", "execution_count": 27, - "id": "d604ca70-e0af-4cdb-8046-53e37f4b6afe", + "id": "d1a7be62", "metadata": {}, "outputs": [], "source": [ - "reloaded_model = tf.keras.models.load_model('my_pet_classifier')" + "reloaded_model = tf.keras.models.load_model('my_pet_classifier.keras')" ] }, { "cell_type": "code", "execution_count": 28, - "id": "7b8be0a1-e16b-4509-8cbc-357def8bb282", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[,\n", - " ,\n", - " ,\n", - " ,\n", - " ,\n", - " ,\n", - " ,\n", - " ,\n", - " ,\n", - " ,\n", - " ,\n", - " ,\n", - " ]" - ] - }, - "execution_count": 28, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "reloaded_model.inputs" - ] - }, - { - "cell_type": "code", - "execution_count": 29, "id": "f3d2a2d5-fd4d-4320-bacc-fd4571cec709", "metadata": {}, "outputs": [ @@ -775,8 +772,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "1/1 [==============================] - 0s 264ms/step\n", - "This particular pet had a 76.8 percent probability of getting adopted.\n" + "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 26ms/step\n", + "This particular pet had a 81.1 percent probability of getting adopted.\n" ] } ], @@ -817,17 +814,58 @@ }, { "cell_type": "code", - "execution_count": 30, - "id": "3c64fd7b-3d1e-40f8-ab64-b5c13f8bbe77", + "execution_count": 29, + "id": "fc8a0536", "metadata": {}, "outputs": [], "source": [ - "df = spark.createDataFrame(dataframe)" + "from pyspark import SparkConf\n", + "from pyspark.sql import SparkSession" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "60dff1da", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" ] }, { "cell_type": "code", "execution_count": 31, + "id": "3c64fd7b-3d1e-40f8-ab64-b5c13f8bbe77", + "metadata": {}, + "outputs": [], + "source": [ + "df = spark.createDataFrame(dataframe).repartition(8)" + ] + }, + { + "cell_type": "code", + "execution_count": 32, "id": "1be8215b-5068-41b4-849c-1c3ea7bb108a", "metadata": {}, "outputs": [ @@ -845,7 +883,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 33, "id": "d4dbde99-cf65-4c15-a163-754a0201a48d", "metadata": {}, "outputs": [ @@ -896,7 +934,7 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 34, "id": "4c21296c-20ed-43f8-921a-c85a820d1819", "metadata": {}, "outputs": [], @@ -911,7 +949,7 @@ }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 35, "id": "04b38f3a-70ea-4746-9f52-c50087401508", "metadata": {}, "outputs": [ @@ -919,28 +957,28 @@ "name": "stdout", "output_type": "stream", "text": [ - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", - "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target|\n", - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", - "| Dog| 7| Spitz| Male| Brown| Golden| Medium| Medium| Not Sure| No|Healthy| 0| 2| 0|\n", - "| Cat| 4|Domestic Short Hair|Female| White|No Color| Medium| Short| No| No|Healthy| 0| 2| 1|\n", - "| Cat| 3|Domestic Short Hair| Male| Brown|No Color| Small| Short| No| No|Healthy| 0| 2| 1|\n", - "| Cat| 11|Domestic Short Hair|Female| Brown| Yellow| Medium| Short| Yes| Yes|Healthy|100| 1| 0|\n", - "| Dog| 1| Mixed Breed| Male| Black| Brown| Medium| Medium| No| No|Healthy| 0| 1| 1|\n", - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", + "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target|\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", + "| Cat| 1|Domestic Medium Hair|Female| White|No Color| Small| Medium| No| No|Healthy| 0| 2| 1|\n", + "| Dog| 2| Mixed Breed|Female| Black| Brown| Medium| Medium| No| No|Healthy| 0| 3| 1|\n", + "| Dog| 18| Dalmatian|Female| Black| White| Medium| Medium| Yes| No|Healthy|350| 5| 1|\n", + "| Dog| 3| Mixed Breed|Female| Black|No Color| Medium| Short| No| No|Healthy| 0| 1| 0|\n", + "| Dog| 2| Mixed Breed| Male| Black| Brown| Medium| Short| No| No|Healthy| 0| 1| 1|\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", "only showing top 5 rows\n", "\n" ] } ], "source": [ - "df = spark.read.parquet(\"datasets/petfinder-mini\")\n", + "df = spark.read.parquet(\"datasets/petfinder-mini\").cache()\n", "df.show(5)" ] }, { "cell_type": "code", - "execution_count": 35, + "execution_count": 36, "id": "29c27243-7c74-4045-aaf1-f75a322c0530", "metadata": {}, "outputs": [ @@ -959,7 +997,7 @@ }, { "cell_type": "code", - "execution_count": 36, + "execution_count": 37, "id": "47508b14-97fa-42ee-a7d0-6175e6408283", "metadata": { "tags": [] @@ -981,18 +1019,18 @@ }, { "cell_type": "code", - "execution_count": 37, + "execution_count": 38, "id": "d62eb95a-54c6-44d2-9279-38fb65e0e160", "metadata": {}, "outputs": [], "source": [ "# get absolute path to model\n", - "model_dir = \"{}/my_pet_classifier\".format(os.getcwd())" + "model_dir = \"{}/my_pet_classifier.keras\".format(os.getcwd())" ] }, { "cell_type": "code", - "execution_count": 38, + "execution_count": 39, "id": "45665acf-50c8-445b-a985-b3dabd734709", "metadata": {}, "outputs": [], @@ -1000,6 +1038,16 @@ "def predict_batch_fn():\n", " import tensorflow as tf\n", " import pandas as pd\n", + " \n", + " # Enable GPU memory growth to avoid CUDA OOM\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + "\n", " model = tf.keras.models.load_model(model_dir)\n", "\n", " def predict(t, a, b, g, c1, c2, m, f, v, s, h, fee, p):\n", @@ -1026,7 +1074,7 @@ }, { "cell_type": "code", - "execution_count": 39, + "execution_count": 40, "id": "815e3b5f-7914-4235-85fa-50153dcd3d30", "metadata": {}, "outputs": [], @@ -1039,7 +1087,7 @@ }, { "cell_type": "code", - "execution_count": 40, + "execution_count": 41, "id": "da03a0c6-2d39-425e-a9fa-57c139cca1ed", "metadata": {}, "outputs": [ @@ -1047,16 +1095,23 @@ "name": "stderr", "output_type": "stream", "text": [ - "23/05/19 17:53:03 WARN package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.\n", - " \r" + "24/10/03 17:39:09 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.\n", + "[Stage 4:> (0 + 8) / 8]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 271 ms, sys: 16.3 ms, total: 288 ms\n", - "Wall time: 7.58 s\n" + "CPU times: user 24.1 ms, sys: 6.37 ms, total: 30.4 ms\n", + "Wall time: 4.58 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" ] } ], @@ -1068,7 +1123,7 @@ }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 42, "id": "03990c76-7198-49a7-bb5d-6870be915fb3", "metadata": {}, "outputs": [ @@ -1076,15 +1131,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 5:====================================> (5 + 3) / 8]\r" + "[Stage 5:> (0 + 8) / 8]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 71.2 ms, sys: 801 µs, total: 72 ms\n", - "Wall time: 2.41 s\n" + "CPU times: user 98.2 ms, sys: 8.34 ms, total: 107 ms\n", + "Wall time: 1.57 s\n" ] }, { @@ -1103,7 +1158,7 @@ }, { "cell_type": "code", - "execution_count": 42, + "execution_count": 43, "id": "edb93cf3-c248-40c9-b8dc-acc8f51786a9", "metadata": {}, "outputs": [ @@ -1111,15 +1166,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 6:====================================> (5 + 3) / 8]\r" + "[Stage 6:> (0 + 8) / 8]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 84.9 ms, sys: 0 ns, total: 84.9 ms\n", - "Wall time: 2.41 s\n" + "CPU times: user 17.6 ms, sys: 2.19 ms, total: 19.8 ms\n", + "Wall time: 1.51 s\n" ] }, { @@ -1138,7 +1193,7 @@ }, { "cell_type": "code", - "execution_count": 43, + "execution_count": 44, "id": "a91f19cb-f7f1-4669-aff1-be594bea5378", "metadata": { "scrolled": true @@ -1148,30 +1203,30 @@ "name": "stdout", "output_type": "stream", "text": [ - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+----------+\n", - "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target| preds|\n", - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+----------+\n", - "| Dog| 7| Spitz| Male| Brown| Golden| Medium| Medium| Not Sure| No|Healthy| 0| 2| 0| 1.2876476|\n", - "| Cat| 4|Domestic Short Hair|Female| White|No Color| Medium| Short| No| No|Healthy| 0| 2| 1| 0.8216562|\n", - "| Cat| 3|Domestic Short Hair| Male| Brown|No Color| Small| Short| No| No|Healthy| 0| 2| 1| 1.1362813|\n", - "| Cat| 11|Domestic Short Hair|Female| Brown| Yellow| Medium| Short| Yes| Yes|Healthy|100| 1| 0|-0.5538333|\n", - "| Dog| 1| Mixed Breed| Male| Black| Brown| Medium| Medium| No| No|Healthy| 0| 1| 1| 1.8639014|\n", - "| Dog| 12|German Shepherd Dog|Female| Black| Brown| Large| Medium| Not Sure| Not Sure|Healthy| 0| 4| 1| 0.646166|\n", - "| Cat| 3|Oriental Short Hair|Female| White|No Color| Medium| Short| Not Sure| Not Sure|Healthy| 0| 3| 1| 1.2250762|\n", - "| Cat| 6| Russian Blue| Male| Gray|No Color| Medium| Short| Not Sure| No|Healthy| 0| 3| 1| 1.2427491|\n", - "| Dog| 13| Miniature Pinscher|Female|Golden|No Color| Small| Short| Yes| Yes|Healthy| 0| 6| 1| 1.2280617|\n", - "| Dog| 5| Mixed Breed| Male| Black| White| Large| Medium| Yes| Yes|Healthy| 0| 6| 1| 0.3791974|\n", - "| Dog| 62| Golden Retriever|Female|Golden| Cream| Medium| Medium| Yes| Yes|Healthy| 0| 3| 1| 0.9814444|\n", - "| Cat| 2| Domestic Long Hair|Female| Cream|No Color| Medium| Medium| No| No|Healthy| 80| 5| 1| 2.7954462|\n", - "| Cat| 12|Domestic Short Hair|Female| White|No Color| Medium| Short| Yes| Yes|Healthy| 0| 1| 1|-0.2367835|\n", - "| Dog| 1| Mixed Breed|Female| Brown|No Color| Medium| Medium| No| No|Healthy| 0| 3| 1| 1.8235079|\n", - "| Dog| 5| Mixed Breed|Female| Brown|No Color| Medium| Medium| Yes| No|Healthy| 0| 1| 1|0.16853562|\n", - "| Dog| 2| Mixed Breed|Female| Black| Brown| Medium| Medium| Yes| No|Healthy| 0| 1| 0| 1.3014978|\n", - "| Cat| 4|Domestic Short Hair| Male| Black|No Color| Medium| Short| No| No|Healthy| 0| 3| 0| 1.0102129|\n", - "| Dog| 2| Mixed Breed| Male| Cream|No Color| Medium| Medium| No| No|Healthy| 0| 3| 1| 2.032087|\n", - "| Dog| 24| Pomeranian| Male|Yellow| White| Small| Medium| Not Sure| Not Sure|Healthy| 0| 9| 1| 1.7647744|\n", - "| Cat| 36|Domestic Short Hair|Female| Gray| White| Small| Short| Not Sure| Not Sure|Healthy| 0| 2| 1|0.16341272|\n", - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+----------+\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+------------+---+--------+------+-----------+\n", + "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target| preds|\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+------------+---+--------+------+-----------+\n", + "| Cat| 1|Domestic Medium Hair|Female| White|No Color| Small| Medium| No| No| Healthy| 0| 2| 1| 1.9543833|\n", + "| Dog| 2| Mixed Breed|Female| Black| Brown| Medium| Medium| No| No| Healthy| 0| 3| 1| 1.7454995|\n", + "| Dog| 18| Dalmatian|Female| Black| White| Medium| Medium| Yes| No| Healthy|350| 5| 1| 1.5183508|\n", + "| Dog| 3| Mixed Breed|Female| Black|No Color| Medium| Short| No| No| Healthy| 0| 1| 0| 0.67013955|\n", + "| Dog| 2| Mixed Breed| Male| Black| Brown| Medium| Short| No| No| Healthy| 0| 1| 1| 1.5492269|\n", + "| Dog| 36| Mixed Breed| Male| Brown|No Color| Medium| Medium| Not Sure| Yes|Minor Injury| 0| 1| 0|-0.27595556|\n", + "| Cat| 6| Domestic Short Hair|Female| Black| White| Medium| Short| Yes| No| Healthy| 0| 5| 0| 0.7229555|\n", + "| Dog| 72| Golden Retriever|Female| Cream|No Color| Large| Medium| Yes| Not Sure| Healthy| 0| 1| 1| 0.7397226|\n", + "| Cat| 2| Domestic Short Hair| Male| Black| White| Small| Short| No| Yes| Healthy| 0| 4| 1| 1.4016482|\n", + "| Dog| 3| Irish Terrier| Male| Brown| Cream| Medium| Medium| Yes| No|Minor Injury|200| 3| 0| 1.3436754|\n", + "| Dog| 2| Mixed Breed|Female| White|No Color| Medium| Medium| Yes| No| Healthy| 0| 1| 1| 1.156439|\n", + "| Dog| 2| Mixed Breed| Male| Brown|No Color| Medium| Medium| Yes| No| Healthy| 0| 4| 1| 1.7760799|\n", + "| Cat| 2| Domestic Short Hair| Male| Black| Gray| Medium| Short| No| No| Healthy| 0| 2| 1| 1.8319463|\n", + "| Dog| 1| German Shepherd Dog| Male| Gray|No Color| Medium| Medium| No| No| Healthy| 0| 6| 1| 2.5471144|\n", + "| Dog| 24| Golden Retriever| Male|Yellow| Cream| Medium| Long| Yes| Yes| Healthy| 0| 7| 1| 1.4675076|\n", + "| Dog| 1| Mixed Breed|Female| Black| Brown| Small| Medium| No| Yes| Healthy| 0| 1| 0| 0.8451028|\n", + "| Cat| 12| Tuxedo| Male| Black|No Color| Small| Medium| Yes| Yes| Healthy| 50| 1| 1| 0.6487097|\n", + "| Cat| 3| Domestic Short Hair|Female| Black|No Color| Small| Short| No| No| Healthy| 0| 1| 1| 1.0688435|\n", + "| Dog| 2| Mixed Breed|Female| Brown| White| Medium| Short| No| No| Healthy| 0| 1| 1| 1.4086031|\n", + "| Dog| 11| Mixed Breed|Female|Golden|No Color| Medium| Short| Yes| Yes| Healthy| 0| 9| 1| 0.28429908|\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+------------+---+--------+------+-----------+\n", "only showing top 20 rows\n", "\n" ] @@ -1196,19 +1251,21 @@ "id": "22d1805b-7cac-4b27-9359-7a25b4ef3f71", "metadata": {}, "source": [ - "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments), using a conda-pack environment created as follows:\n", + "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments) for Triton 24.08, using a conda-pack environment created as follows:\n", "```\n", - "conda create -n tf-gpu -c conda-forge python=3.8\n", + "conda create -n tf-gpu -c conda-forge python=3.10.0\n", "conda activate tf-gpu\n", - "pip install tensorflow\n", - "pip install conda-pack\n", - "conda pack # tf-gpu.tar.gz\n", + "\n", + "export PYTHONNOUSERSITE=True\n", + "pip install numpy==1.26.4 tensorflow[and-cuda] conda-pack\n", + "\n", + "conda-pack # tf-gpu.tar.gz\n", "```" ] }, { "cell_type": "code", - "execution_count": 44, + "execution_count": 45, "id": "2605d134-ef75-4d94-9b16-2c6d85f29bef", "metadata": {}, "outputs": [], @@ -1222,14 +1279,23 @@ }, { "cell_type": "code", - "execution_count": 45, + "execution_count": 46, "id": "4666e618-8038-4dc5-9be7-793aedbf4500", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper\n", + "sudo: a password is required\n" + ] + } + ], "source": [ "%%bash\n", "# copy custom model to expected layout for Triton\n", - "rm -rf models\n", + "sudo rm -rf models\n", "mkdir -p models\n", "cp -r models_config/feature_columns models\n", "\n", @@ -1247,7 +1313,7 @@ }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 47, "id": "a7fb146c-5319-4831-85f7-f2f3c084b042", "metadata": { "scrolled": true @@ -1266,7 +1332,7 @@ "[True]" ] }, - "execution_count": 46, + "execution_count": 47, "metadata": {}, "output_type": "execute_result" } @@ -1274,7 +1340,7 @@ "source": [ "num_executors = 1\n", "triton_models_dir = \"{}/models\".format(os.getcwd())\n", - "my_pet_classifier_dir = \"{}/my_pet_classifier\".format(os.getcwd())\n", + "my_pet_classifier_dir = \"{}/my_pet_classifier.keras\".format(os.getcwd())\n", "nodeRDD = sc.parallelize(list(range(num_executors)), num_executors)\n", "\n", "def start_triton(it):\n", @@ -1288,7 +1354,7 @@ " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", " else:\n", " container=client.containers.run(\n", - " \"nvcr.io/nvidia/tritonserver:23.04-py3\", \"tritonserver --model-repository=/models\",\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", " detach=True,\n", " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", " name=\"spark-triton\",\n", @@ -1297,7 +1363,7 @@ " shm_size=\"128M\",\n", " volumes={\n", " triton_models_dir: {\"bind\": \"/models\", \"mode\": \"ro\"},\n", - " my_pet_classifier_dir: {\"bind\": \"/my_pet_classifier\", \"mode\": \"ro\"}\n", + " my_pet_classifier_dir: {\"bind\": \"/my_pet_classifier.keras\", \"mode\": \"ro\"}\n", " }\n", " )\n", " print(\">>>> starting triton: {}\".format(container.short_id))\n", @@ -1327,7 +1393,7 @@ }, { "cell_type": "code", - "execution_count": 47, + "execution_count": 48, "id": "fe8dc3e6-f1b1-4a24-85f4-0a5ecabef4c5", "metadata": {}, "outputs": [], @@ -1337,7 +1403,7 @@ }, { "cell_type": "code", - "execution_count": 48, + "execution_count": 49, "id": "ce92f041-930f-48ed-9a03-19f6c249ca27", "metadata": {}, "outputs": [ @@ -1345,15 +1411,15 @@ "name": "stdout", "output_type": "stream", "text": [ - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", - "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target|\n", - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", - "| Dog| 7| Spitz| Male| Brown| Golden| Medium| Medium| Not Sure| No|Healthy| 0| 2| 0|\n", - "| Cat| 4|Domestic Short Hair|Female| White|No Color| Medium| Short| No| No|Healthy| 0| 2| 1|\n", - "| Cat| 3|Domestic Short Hair| Male| Brown|No Color| Small| Short| No| No|Healthy| 0| 2| 1|\n", - "| Cat| 11|Domestic Short Hair|Female| Brown| Yellow| Medium| Short| Yes| Yes|Healthy|100| 1| 0|\n", - "| Dog| 1| Mixed Breed| Male| Black| Brown| Medium| Medium| No| No|Healthy| 0| 1| 1|\n", - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", + "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target|\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", + "| Cat| 1|Domestic Medium Hair|Female| White|No Color| Small| Medium| No| No|Healthy| 0| 2| 1|\n", + "| Dog| 2| Mixed Breed|Female| Black| Brown| Medium| Medium| No| No|Healthy| 0| 3| 1|\n", + "| Dog| 18| Dalmatian|Female| Black| White| Medium| Medium| Yes| No|Healthy|350| 5| 1|\n", + "| Dog| 3| Mixed Breed|Female| Black|No Color| Medium| Short| No| No|Healthy| 0| 1| 0|\n", + "| Dog| 2| Mixed Breed| Male| Black| Brown| Medium| Short| No| No|Healthy| 0| 1| 1|\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+\n", "only showing top 5 rows\n", "\n" ] @@ -1365,7 +1431,7 @@ }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 50, "id": "4cfb3f34-a215-4781-91bf-2bec85e15633", "metadata": {}, "outputs": [ @@ -1384,7 +1450,7 @@ }, { "cell_type": "code", - "execution_count": 50, + "execution_count": 51, "id": "b315ee72-62af-476b-a994-0dba72d5f96e", "metadata": { "scrolled": true, @@ -1407,7 +1473,7 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 52, "id": "da004eca-f7ad-4ee3-aa88-a6a20c1b72e5", "metadata": {}, "outputs": [], @@ -1417,7 +1483,7 @@ " import tritonclient.grpc as grpcclient\n", " \n", " np_types = {\n", - " \"BOOL\": np.dtype(np.bool8),\n", + " \"BOOL\": np.dtype(np.bool_),\n", " \"INT8\": np.dtype(np.int8),\n", " \"INT16\": np.dtype(np.int16),\n", " \"INT32\": np.dtype(np.int32),\n", @@ -1477,7 +1543,7 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 53, "id": "2ffb020e-dc93-456b-bee6-405611eee1e1", "metadata": {}, "outputs": [], @@ -1493,7 +1559,7 @@ }, { "cell_type": "code", - "execution_count": 53, + "execution_count": 54, "id": "7657f820-5ec2-4ac8-a107-4b58773d204a", "metadata": {}, "outputs": [ @@ -1501,30 +1567,30 @@ "name": "stdout", "output_type": "stream", "text": [ - "+----+---+----------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+----------+\n", - "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target| preds|\n", - "+----+---+----------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+----------+\n", - "| Dog| 7| Spitz| Male| Brown| Golden| Medium| Medium| Not Sure| No|Healthy| 0| 2| 0| 1.2876476|\n", - "| Cat| 4|Domesti...|Female| White|No Color| Medium| Short| No| No|Healthy| 0| 2| 1| 0.8216562|\n", - "| Cat| 3|Domesti...| Male| Brown|No Color| Small| Short| No| No|Healthy| 0| 2| 1| 1.1362813|\n", - "| Cat| 11|Domesti...|Female| Brown| Yellow| Medium| Short| Yes| Yes|Healthy|100| 1| 0|-0.5538333|\n", - "| Dog| 1|Mixed B...| Male| Black| Brown| Medium| Medium| No| No|Healthy| 0| 1| 1| 1.8639014|\n", - "| Dog| 12|German ...|Female| Black| Brown| Large| Medium| Not Sure| Not Sure|Healthy| 0| 4| 1| 0.646166|\n", - "| Cat| 3|Orienta...|Female| White|No Color| Medium| Short| Not Sure| Not Sure|Healthy| 0| 3| 1| 1.2250762|\n", - "| Cat| 6|Russian...| Male| Gray|No Color| Medium| Short| Not Sure| No|Healthy| 0| 3| 1| 1.2427491|\n", - "| Dog| 13|Miniatu...|Female|Golden|No Color| Small| Short| Yes| Yes|Healthy| 0| 6| 1| 1.2280617|\n", - "| Dog| 5|Mixed B...| Male| Black| White| Large| Medium| Yes| Yes|Healthy| 0| 6| 1| 0.3791974|\n", - "| Dog| 62|Golden ...|Female|Golden| Cream| Medium| Medium| Yes| Yes|Healthy| 0| 3| 1| 0.9814444|\n", - "| Cat| 2|Domesti...|Female| Cream|No Color| Medium| Medium| No| No|Healthy| 80| 5| 1| 2.7954462|\n", - "| Cat| 12|Domesti...|Female| White|No Color| Medium| Short| Yes| Yes|Healthy| 0| 1| 1|-0.2367835|\n", - "| Dog| 1|Mixed B...|Female| Brown|No Color| Medium| Medium| No| No|Healthy| 0| 3| 1| 1.8235079|\n", - "| Dog| 5|Mixed B...|Female| Brown|No Color| Medium| Medium| Yes| No|Healthy| 0| 1| 1|0.16853562|\n", - "| Dog| 2|Mixed B...|Female| Black| Brown| Medium| Medium| Yes| No|Healthy| 0| 1| 0| 1.3014978|\n", - "| Cat| 4|Domesti...| Male| Black|No Color| Medium| Short| No| No|Healthy| 0| 3| 0| 1.0102129|\n", - "| Dog| 2|Mixed B...| Male| Cream|No Color| Medium| Medium| No| No|Healthy| 0| 3| 1| 2.032087|\n", - "| Dog| 24|Pomeranian| Male|Yellow| White| Small| Medium| Not Sure| Not Sure|Healthy| 0| 9| 1| 1.7647744|\n", - "| Cat| 36|Domesti...|Female| Gray| White| Small| Short| Not Sure| Not Sure|Healthy| 0| 2| 1|0.16341272|\n", - "+----+---+----------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+----------+\n", + "+----+---+----------+------+------+--------+------------+---------+----------+----------+----------+---+--------+------+----------+\n", + "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target| preds|\n", + "+----+---+----------+------+------+--------+------------+---------+----------+----------+----------+---+--------+------+----------+\n", + "| Cat| 1|Domesti...|Female| White|No Color| Small| Medium| No| No| Healthy| 0| 2| 1| 1.9543833|\n", + "| Dog| 2|Mixed B...|Female| Black| Brown| Medium| Medium| No| No| Healthy| 0| 3| 1| 1.7454995|\n", + "| Dog| 18| Dalmatian|Female| Black| White| Medium| Medium| Yes| No| Healthy|350| 5| 1| 1.5183508|\n", + "| Dog| 3|Mixed B...|Female| Black|No Color| Medium| Short| No| No| Healthy| 0| 1| 0|0.67013955|\n", + "| Dog| 2|Mixed B...| Male| Black| Brown| Medium| Short| No| No| Healthy| 0| 1| 1| 1.5492269|\n", + "| Dog| 36|Mixed B...| Male| Brown|No Color| Medium| Medium| Not Sure| Yes|Minor I...| 0| 1| 0|-0.2759...|\n", + "| Cat| 6|Domesti...|Female| Black| White| Medium| Short| Yes| No| Healthy| 0| 5| 0| 0.7229555|\n", + "| Dog| 72|Golden ...|Female| Cream|No Color| Large| Medium| Yes| Not Sure| Healthy| 0| 1| 1| 0.7397226|\n", + "| Cat| 2|Domesti...| Male| Black| White| Small| Short| No| Yes| Healthy| 0| 4| 1| 1.4016482|\n", + "| Dog| 3|Irish T...| Male| Brown| Cream| Medium| Medium| Yes| No|Minor I...|200| 3| 0| 1.3436754|\n", + "| Dog| 2|Mixed B...|Female| White|No Color| Medium| Medium| Yes| No| Healthy| 0| 1| 1| 1.156439|\n", + "| Dog| 2|Mixed B...| Male| Brown|No Color| Medium| Medium| Yes| No| Healthy| 0| 4| 1| 1.7760799|\n", + "| Cat| 2|Domesti...| Male| Black| Gray| Medium| Short| No| No| Healthy| 0| 2| 1| 1.8319463|\n", + "| Dog| 1|German ...| Male| Gray|No Color| Medium| Medium| No| No| Healthy| 0| 6| 1| 2.5471144|\n", + "| Dog| 24|Golden ...| Male|Yellow| Cream| Medium| Long| Yes| Yes| Healthy| 0| 7| 1| 1.4675076|\n", + "| Dog| 1|Mixed B...|Female| Black| Brown| Small| Medium| No| Yes| Healthy| 0| 1| 0| 0.8451028|\n", + "| Cat| 12| Tuxedo| Male| Black|No Color| Small| Medium| Yes| Yes| Healthy| 50| 1| 1| 0.6487097|\n", + "| Cat| 3|Domesti...|Female| Black|No Color| Small| Short| No| No| Healthy| 0| 1| 1| 1.0688435|\n", + "| Dog| 2|Mixed B...|Female| Brown| White| Medium| Short| No| No| Healthy| 0| 1| 1| 1.4086031|\n", + "| Dog| 11|Mixed B...|Female|Golden|No Color| Medium| Short| Yes| Yes| Healthy| 0| 9| 1|0.28429908|\n", + "+----+---+----------+------+------+--------+------------+---------+----------+----------+----------+---+--------+------+----------+\n", "only showing top 20 rows\n", "\n" ] @@ -1537,7 +1603,7 @@ }, { "cell_type": "code", - "execution_count": 54, + "execution_count": 55, "id": "e6ff0356-becd-421f-aebb-272497d5ad6a", "metadata": {}, "outputs": [ @@ -1552,8 +1618,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 62.9 ms, sys: 15.5 ms, total: 78.4 ms\n", - "Wall time: 2.62 s\n" + "CPU times: user 17.2 ms, sys: 2.85 ms, total: 20 ms\n", + "Wall time: 2.5 s\n" ] } ], @@ -1565,7 +1631,7 @@ }, { "cell_type": "code", - "execution_count": 55, + "execution_count": 56, "id": "ce18ee7c-5958-4986-b200-6d986fcc6243", "metadata": {}, "outputs": [ @@ -1580,8 +1646,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 70 ms, sys: 6.89 ms, total: 76.9 ms\n", - "Wall time: 2.51 s\n" + "CPU times: user 23.4 ms, sys: 984 μs, total: 24.4 ms\n", + "Wall time: 2.5 s\n" ] }, { @@ -1600,7 +1666,7 @@ }, { "cell_type": "code", - "execution_count": 56, + "execution_count": 57, "id": "0888ce40-b2c4-4aed-8ccb-6a8bcd00abc8", "metadata": { "tags": [] @@ -1610,22 +1676,15 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 14:===========================================> (6 + 2) / 8]\r" + " \r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 85.5 ms, sys: 6.85 ms, total: 92.4 ms\n", - "Wall time: 2.3 s\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" + "CPU times: user 14.7 ms, sys: 4.61 ms, total: 19.3 ms\n", + "Wall time: 2.47 s\n" ] } ], @@ -1637,7 +1696,7 @@ }, { "cell_type": "code", - "execution_count": 57, + "execution_count": 58, "id": "d45812b5-f584-41a4-a821-2b59e065671c", "metadata": {}, "outputs": [ @@ -1645,30 +1704,30 @@ "name": "stdout", "output_type": "stream", "text": [ - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+----------+\n", - "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target| preds|\n", - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+----------+\n", - "| Dog| 7| Spitz| Male| Brown| Golden| Medium| Medium| Not Sure| No|Healthy| 0| 2| 0| 1.2876476|\n", - "| Cat| 4|Domestic Short Hair|Female| White|No Color| Medium| Short| No| No|Healthy| 0| 2| 1| 0.8216562|\n", - "| Cat| 3|Domestic Short Hair| Male| Brown|No Color| Small| Short| No| No|Healthy| 0| 2| 1| 1.1362813|\n", - "| Cat| 11|Domestic Short Hair|Female| Brown| Yellow| Medium| Short| Yes| Yes|Healthy|100| 1| 0|-0.5538333|\n", - "| Dog| 1| Mixed Breed| Male| Black| Brown| Medium| Medium| No| No|Healthy| 0| 1| 1| 1.8639014|\n", - "| Dog| 12|German Shepherd Dog|Female| Black| Brown| Large| Medium| Not Sure| Not Sure|Healthy| 0| 4| 1| 0.646166|\n", - "| Cat| 3|Oriental Short Hair|Female| White|No Color| Medium| Short| Not Sure| Not Sure|Healthy| 0| 3| 1| 1.2250762|\n", - "| Cat| 6| Russian Blue| Male| Gray|No Color| Medium| Short| Not Sure| No|Healthy| 0| 3| 1| 1.2427491|\n", - "| Dog| 13| Miniature Pinscher|Female|Golden|No Color| Small| Short| Yes| Yes|Healthy| 0| 6| 1| 1.2280617|\n", - "| Dog| 5| Mixed Breed| Male| Black| White| Large| Medium| Yes| Yes|Healthy| 0| 6| 1| 0.3791974|\n", - "| Dog| 62| Golden Retriever|Female|Golden| Cream| Medium| Medium| Yes| Yes|Healthy| 0| 3| 1| 0.9814444|\n", - "| Cat| 2| Domestic Long Hair|Female| Cream|No Color| Medium| Medium| No| No|Healthy| 80| 5| 1| 2.7954462|\n", - "| Cat| 12|Domestic Short Hair|Female| White|No Color| Medium| Short| Yes| Yes|Healthy| 0| 1| 1|-0.2367835|\n", - "| Dog| 1| Mixed Breed|Female| Brown|No Color| Medium| Medium| No| No|Healthy| 0| 3| 1| 1.8235079|\n", - "| Dog| 5| Mixed Breed|Female| Brown|No Color| Medium| Medium| Yes| No|Healthy| 0| 1| 1|0.16853562|\n", - "| Dog| 2| Mixed Breed|Female| Black| Brown| Medium| Medium| Yes| No|Healthy| 0| 1| 0| 1.3014978|\n", - "| Cat| 4|Domestic Short Hair| Male| Black|No Color| Medium| Short| No| No|Healthy| 0| 3| 0| 1.0102129|\n", - "| Dog| 2| Mixed Breed| Male| Cream|No Color| Medium| Medium| No| No|Healthy| 0| 3| 1| 2.032087|\n", - "| Dog| 24| Pomeranian| Male|Yellow| White| Small| Medium| Not Sure| Not Sure|Healthy| 0| 9| 1| 1.7647744|\n", - "| Cat| 36|Domestic Short Hair|Female| Gray| White| Small| Short| Not Sure| Not Sure|Healthy| 0| 2| 1|0.16341272|\n", - "+----+---+-------------------+------+------+--------+------------+---------+----------+----------+-------+---+--------+------+----------+\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+------------+---+--------+------+-----------+\n", + "|Type|Age| Breed1|Gender|Color1| Color2|MaturitySize|FurLength|Vaccinated|Sterilized| Health|Fee|PhotoAmt|target| preds|\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+------------+---+--------+------+-----------+\n", + "| Cat| 1|Domestic Medium Hair|Female| White|No Color| Small| Medium| No| No| Healthy| 0| 2| 1| 1.9543833|\n", + "| Dog| 2| Mixed Breed|Female| Black| Brown| Medium| Medium| No| No| Healthy| 0| 3| 1| 1.7454995|\n", + "| Dog| 18| Dalmatian|Female| Black| White| Medium| Medium| Yes| No| Healthy|350| 5| 1| 1.5183508|\n", + "| Dog| 3| Mixed Breed|Female| Black|No Color| Medium| Short| No| No| Healthy| 0| 1| 0| 0.67013955|\n", + "| Dog| 2| Mixed Breed| Male| Black| Brown| Medium| Short| No| No| Healthy| 0| 1| 1| 1.5492269|\n", + "| Dog| 36| Mixed Breed| Male| Brown|No Color| Medium| Medium| Not Sure| Yes|Minor Injury| 0| 1| 0|-0.27595556|\n", + "| Cat| 6| Domestic Short Hair|Female| Black| White| Medium| Short| Yes| No| Healthy| 0| 5| 0| 0.7229555|\n", + "| Dog| 72| Golden Retriever|Female| Cream|No Color| Large| Medium| Yes| Not Sure| Healthy| 0| 1| 1| 0.7397226|\n", + "| Cat| 2| Domestic Short Hair| Male| Black| White| Small| Short| No| Yes| Healthy| 0| 4| 1| 1.4016482|\n", + "| Dog| 3| Irish Terrier| Male| Brown| Cream| Medium| Medium| Yes| No|Minor Injury|200| 3| 0| 1.3436754|\n", + "| Dog| 2| Mixed Breed|Female| White|No Color| Medium| Medium| Yes| No| Healthy| 0| 1| 1| 1.156439|\n", + "| Dog| 2| Mixed Breed| Male| Brown|No Color| Medium| Medium| Yes| No| Healthy| 0| 4| 1| 1.7760799|\n", + "| Cat| 2| Domestic Short Hair| Male| Black| Gray| Medium| Short| No| No| Healthy| 0| 2| 1| 1.8319463|\n", + "| Dog| 1| German Shepherd Dog| Male| Gray|No Color| Medium| Medium| No| No| Healthy| 0| 6| 1| 2.5471144|\n", + "| Dog| 24| Golden Retriever| Male|Yellow| Cream| Medium| Long| Yes| Yes| Healthy| 0| 7| 1| 1.4675076|\n", + "| Dog| 1| Mixed Breed|Female| Black| Brown| Small| Medium| No| Yes| Healthy| 0| 1| 0| 0.8451028|\n", + "| Cat| 12| Tuxedo| Male| Black|No Color| Small| Medium| Yes| Yes| Healthy| 50| 1| 1| 0.6487097|\n", + "| Cat| 3| Domestic Short Hair|Female| Black|No Color| Small| Short| No| No| Healthy| 0| 1| 1| 1.0688435|\n", + "| Dog| 2| Mixed Breed|Female| Brown| White| Medium| Short| No| No| Healthy| 0| 1| 1| 1.4086031|\n", + "| Dog| 11| Mixed Breed|Female|Golden|No Color| Medium| Short| Yes| Yes| Healthy| 0| 9| 1| 0.28429908|\n", + "+----+---+--------------------+------+------+--------+------------+---------+----------+----------+------------+---+--------+------+-----------+\n", "only showing top 20 rows\n", "\n" ] @@ -1690,7 +1749,7 @@ }, { "cell_type": "code", - "execution_count": 58, + "execution_count": 59, "id": "6914f44f-677f-4db3-be09-783df8d11b8a", "metadata": {}, "outputs": [ @@ -1707,7 +1766,7 @@ "[True]" ] }, - "execution_count": 58, + "execution_count": 59, "metadata": {}, "output_type": "execute_result" } @@ -1731,7 +1790,7 @@ }, { "cell_type": "code", - "execution_count": 59, + "execution_count": 60, "id": "f8c6ee43-8891-4446-986e-1447c5d48bac", "metadata": {}, "outputs": [], @@ -1750,7 +1809,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "spark-dl-tf", "language": "python", "name": "python3" }, @@ -1764,7 +1823,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.10" + "version": "3.11.9" } }, "nbformat": 4, diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/image_classification.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/image_classification.ipynb index 25e809387..5add2686b 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/image_classification.ipynb +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/image_classification.ipynb @@ -11,17 +11,40 @@ "Based on: https://www.tensorflow.org/tutorials/keras/save_and_load" ] }, + { + "cell_type": "markdown", + "id": "5233632d", + "metadata": {}, + "source": [ + "### Using TensorFlow\n", + "Note that cuFFT/cuDNN/cuBLAS registration errors are expected with `tf=2.17.0` and will not affect behavior, as noted in [this issue.](https://github.com/tensorflow/tensorflow/issues/62075) \n", + "This notebook does not demonstrate inference with TensorRT, as [TF-TRT](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#tensorrt-10) does not yet support `tf=2.17.0`. See the `pytorch` notebooks for TensorRT demos." + ] + }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "id": "c8b28f02", "metadata": {}, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:40:20.324462: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", + "2024-10-03 17:40:20.331437: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", + "2024-10-03 17:40:20.339109: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", + "2024-10-03 17:40:20.341362: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n", + "2024-10-03 17:40:20.347337: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", + "To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", + "2024-10-03 17:40:20.672391: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" + ] + }, { "name": "stdout", "output_type": "stream", "text": [ - "2.12.0\n" + "2.17.0\n" ] } ], @@ -30,12 +53,30 @@ "import numpy as np\n", "import subprocess\n", "import tensorflow as tf\n", + "import os\n", "\n", "from tensorflow import keras\n", "\n", "print(tf.version.VERSION)" ] }, + { + "cell_type": "code", + "execution_count": 3, + "id": "e2e67086", + "metadata": {}, + "outputs": [], + "source": [ + "# Enable GPU memory growth\n", + "gpus = tf.config.experimental.list_physical_devices('GPU')\n", + "if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)" + ] + }, { "cell_type": "markdown", "id": "7e0c7ad6", @@ -46,17 +87,31 @@ }, { "cell_type": "code", - "execution_count": 2, + "execution_count": 4, "id": "5b007f7c", "metadata": {}, "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m11490434/11490434\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 0us/step\n" + ] + }, { "data": { "text/plain": [ "((60000, 28, 28), (10000, 28, 28))" ] }, - "execution_count": 2, + "execution_count": 4, "metadata": {}, "output_type": "execute_result" } @@ -69,34 +124,28 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 5, "id": "7b7cedd1", "metadata": {}, - "outputs": [], - "source": [ - "# flatten and normalize\n", - "train_images = train_images.reshape(-1, 784) / 255.0\n", - "test_images = test_images.reshape(-1, 784) / 255.0" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "e77bfbd7", - "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "((60000, 784), (10000, 784))" + "((1000, 784), (1000, 784))" ] }, - "execution_count": 4, + "execution_count": 5, "metadata": {}, "output_type": "execute_result" } ], "source": [ + "train_labels = train_labels[:1000]\n", + "test_labels = test_labels[:1000]\n", + "\n", + "train_images = train_images[:1000].reshape(-1, 28 * 28) / 255.0\n", + "test_images = test_images[:1000].reshape(-1, 28 * 28) / 255.0\n", + "\n", "train_images.shape, test_images.shape" ] }, @@ -110,44 +159,113 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 6, "id": "746d94db", "metadata": {}, "outputs": [ { - "name": "stdout", + "name": "stderr", "output_type": "stream", "text": [ - "Model: \"sequential\"\n", - "_________________________________________________________________\n", - " Layer (type) Output Shape Param # \n", - "=================================================================\n", - " dense (Dense) (None, 512) 401920 \n", - " \n", - " dropout (Dropout) (None, 512) 0 \n", - " \n", - " dense_1 (Dense) (None, 10) 5130 \n", - " \n", - "=================================================================\n", - "Total params: 407,050\n", - "Trainable params: 407,050\n", - "Non-trainable params: 0\n", - "_________________________________________________________________\n" + "/home/rishic/anaconda3/envs/spark-dl-tf/lib/python3.11/site-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n", + " super().__init__(activity_regularizer=activity_regularizer, **kwargs)\n", + "2024-10-03 17:40:21.624052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2021] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 45743 MB memory: -> device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:01:00.0, compute capability: 8.6\n" ] + }, + { + "data": { + "text/html": [ + "
Model: \"sequential\"\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1mModel: \"sequential\"\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
+       "┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃\n",
+       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
+       "│ dense (Dense)                   │ (None, 512)            │       401,920 │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ dropout (Dropout)               │ (None, 512)            │             0 │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ dense_1 (Dense)                 │ (None, 10)             │         5,130 │\n",
+       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
+       "
\n" + ], + "text/plain": [ + "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n", + "┃\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0m┃\n", + "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n", + "│ dense (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m512\u001b[0m) │ \u001b[38;5;34m401,920\u001b[0m │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ dropout (\u001b[38;5;33mDropout\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m512\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ dense_1 (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m10\u001b[0m) │ \u001b[38;5;34m5,130\u001b[0m │\n", + "└─────────────────────────────────┴────────────────────────┴───────────────┘\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Total params: 407,050 (1.55 MB)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m407,050\u001b[0m (1.55 MB)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Trainable params: 407,050 (1.55 MB)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m407,050\u001b[0m (1.55 MB)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Non-trainable params: 0 (0.00 B)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" } ], "source": [ "# Define a simple sequential model\n", "def create_model():\n", - " model = tf.keras.models.Sequential([\n", - " keras.layers.Dense(512, activation='relu', input_shape=(784,)),\n", - " keras.layers.Dropout(0.2),\n", - " keras.layers.Dense(10)\n", + " model = tf.keras.Sequential([\n", + " keras.layers.Dense(512, activation='relu', input_shape=(784,)),\n", + " keras.layers.Dropout(0.2),\n", + " keras.layers.Dense(10)\n", " ])\n", "\n", " model.compile(optimizer='adam',\n", - " loss=tf.losses.SparseCategoricalCrossentropy(from_logits=True),\n", - " metrics=[tf.metrics.SparseCategoricalAccuracy()])\n", + " loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n", + " metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])\n", "\n", " return model\n", "\n", @@ -163,12 +281,12 @@ "id": "605d082a", "metadata": {}, "source": [ - "### Train model" + "### Save checkpoints during training" ] }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 7, "id": "244746be", "metadata": {}, "outputs": [ @@ -176,55 +294,98 @@ "name": "stdout", "output_type": "stream", "text": [ - "Epoch 1/5\n", - "1875/1875 [==============================] - 10s 5ms/step - loss: 0.2183 - sparse_categorical_accuracy: 0.9349 - val_loss: 0.1183 - val_sparse_categorical_accuracy: 0.9624\n", - "Epoch 2/5\n", - "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0978 - sparse_categorical_accuracy: 0.9698 - val_loss: 0.0826 - val_sparse_categorical_accuracy: 0.9744\n", - "Epoch 3/5\n", - "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0686 - sparse_categorical_accuracy: 0.9785 - val_loss: 0.0761 - val_sparse_categorical_accuracy: 0.9762\n", - "Epoch 4/5\n", - "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0522 - sparse_categorical_accuracy: 0.9831 - val_loss: 0.0670 - val_sparse_categorical_accuracy: 0.9809\n", - "Epoch 5/5\n", - "1875/1875 [==============================] - 10s 5ms/step - loss: 0.0424 - sparse_categorical_accuracy: 0.9861 - val_loss: 0.0676 - val_sparse_categorical_accuracy: 0.9796\n" + "Epoch 1/10\n" ] }, { - "data": { - "text/plain": [ - "" - ] - }, - "execution_count": 6, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "model.fit(train_images, \n", - " train_labels, \n", - " epochs=5,\n", - " validation_data=(test_images, test_labels))" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "6d3bba9e", - "metadata": {}, - "outputs": [ + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", + "I0000 00:00:1727977222.161202 1835280 service.cc:146] XLA service 0x7ec778008e00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n", + "I0000 00:00:1727977222.161216 1835280 service.cc:154] StreamExecutor device (0): NVIDIA RTX A6000, Compute Capability 8.6\n", + "2024-10-03 17:40:22.168848: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n", + "2024-10-03 17:40:22.206298: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 8907\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m24s\u001b[0m 778ms/step - loss: 2.3278 - sparse_categorical_accuracy: 0.1250" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "I0000 00:00:1727977222.715572 1835280 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 17ms/step - loss: 1.5867 - sparse_categorical_accuracy: 0.5096 " + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:40:23.780912: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'gemm_fusion_dot_33', 4 bytes spill stores, 4 bytes spill loads\n", + "\n" + ] + }, { "name": "stdout", "output_type": "stream", "text": [ - "1/1 [==============================] - 0s 55ms/step\n" + "\n", + "Epoch 1: val_sparse_categorical_accuracy improved from -inf to 0.78700, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 44ms/step - loss: 1.5733 - sparse_categorical_accuracy: 0.5144 - val_loss: 0.7061 - val_sparse_categorical_accuracy: 0.7870\n", + "Epoch 2/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 13ms/step - loss: 0.5514 - sparse_categorical_accuracy: 0.8438\n", + "Epoch 2: val_sparse_categorical_accuracy improved from 0.78700 to 0.83700, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step - loss: 0.4276 - sparse_categorical_accuracy: 0.8935 - val_loss: 0.5268 - val_sparse_categorical_accuracy: 0.8370\n", + "Epoch 3/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 20ms/step - loss: 0.1458 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 3: val_sparse_categorical_accuracy improved from 0.83700 to 0.85600, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.2721 - sparse_categorical_accuracy: 0.9236 - val_loss: 0.4716 - val_sparse_categorical_accuracy: 0.8560\n", + "Epoch 4/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 13ms/step - loss: 0.2223 - sparse_categorical_accuracy: 0.9375\n", + "Epoch 4: val_sparse_categorical_accuracy did not improve from 0.85600\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.2159 - sparse_categorical_accuracy: 0.9547 - val_loss: 0.4682 - val_sparse_categorical_accuracy: 0.8540\n", + "Epoch 5/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 16ms/step - loss: 0.1483 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 5: val_sparse_categorical_accuracy improved from 0.85600 to 0.86900, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step - loss: 0.1457 - sparse_categorical_accuracy: 0.9716 - val_loss: 0.4285 - val_sparse_categorical_accuracy: 0.8690\n", + "Epoch 6/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 13ms/step - loss: 0.0836 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 6: val_sparse_categorical_accuracy did not improve from 0.86900\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.1292 - sparse_categorical_accuracy: 0.9712 - val_loss: 0.4551 - val_sparse_categorical_accuracy: 0.8580\n", + "Epoch 7/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 12ms/step - loss: 0.0920 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 7: val_sparse_categorical_accuracy did not improve from 0.86900\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.0974 - sparse_categorical_accuracy: 0.9822 - val_loss: 0.4016 - val_sparse_categorical_accuracy: 0.8670\n", + "Epoch 8/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 12ms/step - loss: 0.0993 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 8: val_sparse_categorical_accuracy did not improve from 0.86900\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.0702 - sparse_categorical_accuracy: 0.9920 - val_loss: 0.3999 - val_sparse_categorical_accuracy: 0.8650\n", + "Epoch 9/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 12ms/step - loss: 0.0599 - sparse_categorical_accuracy: 1.0000\n", + "Epoch 9: val_sparse_categorical_accuracy improved from 0.86900 to 0.87800, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.0457 - sparse_categorical_accuracy: 0.9974 - val_loss: 0.4145 - val_sparse_categorical_accuracy: 0.8780\n", + "Epoch 10/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 13ms/step - loss: 0.0286 - sparse_categorical_accuracy: 1.0000\n", + "Epoch 10: val_sparse_categorical_accuracy did not improve from 0.87800\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.0351 - sparse_categorical_accuracy: 0.9987 - val_loss: 0.4200 - val_sparse_categorical_accuracy: 0.8720\n" ] }, { "data": { "text/plain": [ - "array([[ -9.547884 , -4.7447677, -0.956994 , 1.0100659, -15.265252 ,\n", - " -7.811299 , -15.518889 , 14.113535 , -4.9397273, -4.3322315]],\n", - " dtype=float32)" + "" ] }, "execution_count": 7, @@ -233,230 +394,273 @@ } ], "source": [ - "test_img = test_images[:1]\n", - "prediction = model.predict(test_img)\n", - "prediction" + "checkpoint_path = \"training_1/checkpoint.model.keras\"\n", + "checkpoint_dir = os.path.dirname(checkpoint_path)\n", + "\n", + "# Create a callback that saves the model's weights\n", + "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,\n", + " monitor='val_sparse_categorical_accuracy',\n", + " mode='max',\n", + " save_best_only=True,\n", + " verbose=1)\n", + "\n", + "# Train the model with the new callback\n", + "model.fit(train_images, \n", + " train_labels, \n", + " epochs=10,\n", + " validation_data=(test_images, test_labels),\n", + " callbacks=[cp_callback]) # Pass callback to training\n", + "\n", + "# This may generate warnings related to saving the state of the optimizer.\n", + "# These warnings (and similar warnings throughout this notebook)\n", + "# are in place to discourage outdated usage, and can be ignored." ] }, { "cell_type": "code", "execution_count": 8, - "id": "e700b66a", + "id": "310eae08", "metadata": {}, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAiqUlEQVR4nO3de3RU9d3v8c/kNtySgZC7BAgooNxcoqQcLkaJCUFZIPRUvHQBywJiQCGP1calIJZ10mJLUUvBP1qoFcTyKHCkFAUkQWygDwgPBytZkBUFFklAjslAgHCZ3/mDw9QhCbDDDL8kvF9r7bWYvX/fvb/ZbPhkz96zx2WMMQIA4CYLs90AAODWRAABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABDnTt2lUTJ070vy4sLJTL5VJhYWHQtuFyufTaa68FbX1AU0UAodlYtmyZXC6Xf2rVqpV69Oih6dOnq7Ky0nZ7jqxfv77ZhMwP9/mV00MPPWS7PTRjEbYbAJx6/fXXlZaWprNnz2rbtm1avHix1q9fr3379qlNmzY3tZdhw4bpzJkzioqKclS3fv16LVq0qN4QOnPmjCIims4/zb/85S915u3cuVNvvvmmsrKyLHSElqLpHOXAdcrJydG9994rSfrZz36mjh07asGCBVq7dq0ef/zxemtqamrUtm3boPcSFhamVq1aBXWdwV7fjXrqqafqzLv81mND+xu4HrwFh2bvwQcflCSVlZVJkiZOnKh27dqptLRUI0eOVHR0tJ588klJks/n08KFC9W7d2+1atVKiYmJmjp1qr7//vuAdRpjNG/ePHXq1Elt2rTRAw88oK+++qrOthu6BrRjxw6NHDlSHTp0UNu2bdWvXz+9+eab/v4WLVokKfDtrcvquwa0e/du5eTkKCYmRu3atdPw4cO1ffv2gDGX36L84osvlJeXp/j4eLVt21aPPvqojh8/HjC2urpa+/fvV3V19fXs4gC1tbX68MMPdf/996tTp06O64HLOANCs1daWipJ6tixo3/ehQsXlJ2drSFDhug3v/mN/625qVOnatmyZZo0aZKee+45lZWV6fe//712796tL774QpGRkZKk2bNna968eRo5cqRGjhypL7/8UllZWTp37tw1+9m4caMeeeQRJScn6/nnn1dSUpK+/vprrVu3Ts8//7ymTp2qo0ePauPGjfW+vXWlr776SkOHDlVMTIxefPFFRUZG6p133lFGRoaKioqUnp4eMH7GjBnq0KGD5syZo2+++UYLFy7U9OnT9cEHH/jHrF69WpMmTdLSpUsDbqq4HuvXr1dVVZU/1IFGM0AzsXTpUiPJbNq0yRw/ftwcPnzYrFy50nTs2NG0bt3aHDlyxBhjzIQJE4wk84tf/CKg/vPPPzeSzPLlywPmb9iwIWD+sWPHTFRUlHn44YeNz+fzj3v55ZeNJDNhwgT/vC1bthhJZsuWLcYYYy5cuGDS0tJMly5dzPfffx+wnR+uKzc31zT0z0+SmTNnjv/1mDFjTFRUlCktLfXPO3r0qImOjjbDhg2rs38yMzMDtjVr1iwTHh5uqqqq6oxdunRpvT1czbhx44zb7a7z8wFO8RYcmp3MzEzFx8crNTVV48ePV7t27bR69WrddtttAeOmTZsW8HrVqlXyeDx66KGH9N133/mnAQMGqF27dtqyZYskadOmTTp37pxmzJgR8NbYzJkzr9nb7t27VVZWppkzZ6p9+/YBy364rut18eJFffrppxozZoy6devmn5+cnKwnnnhC27Ztk9frDaiZMmVKwLaGDh2qixcv6ttvv/XPmzhxoowxjs9+vF6v/va3v2nkyJF1fj7AKd6CQ7OzaNEi9ejRQxEREUpMTFTPnj0VFhb4u1RERESd6xMHDhxQdXW1EhIS6l3vsWPHJMn/H/Udd9wRsDw+Pl4dOnS4am+X3w7s06fP9f9AV3H8+HGdPn1aPXv2rLPszjvvlM/n0+HDh9W7d2///M6dOweMu9zzlde5GuPDDz/U2bNnefsNQUEAodkZOHCg/y64hrjd7jqh5PP5lJCQoOXLl9dbEx8fH7QebQoPD693vjHmhte9fPlyeTwePfLIIze8LoAAwi2je/fu2rRpkwYPHqzWrVs3OK5Lly6SLp0x/fBtr+PHj1/zLKJ79+6SpH379ikzM7PBcdf7dlx8fLzatGmjkpKSOsv279+vsLAwpaamXte6blR5ebm2bNmiiRMnyu1235RtomXjGhBuGT/5yU908eJF/fKXv6yz7MKFC6qqqpJ06RpTZGSk3n777YCzhoULF15zG/fcc4/S0tK0cOFC//ou++G6Ln8m6coxVwoPD1dWVpbWrl2rb775xj+/srJSK1as0JAhQxQTE3PNvq7UmNuwV65cKZ/Px9tvCBrOgHDLuP/++zV16lQVFBRoz549ysrKUmRkpA4cOKBVq1bpzTff1I9//GPFx8frhRdeUEFBgR555BGNHDlSu3fv1t///nfFxcVddRthYWFavHixRo0apbvvvluTJk1ScnKy9u/fr6+++kqffPKJJGnAgAGSpOeee07Z2dkKDw/X+PHj613nvHnztHHjRg0ZMkTPPvusIiIi9M4776i2tlbz589v1L5ozG3Yy5cvV0pKijIyMhq1TeBKBBBuKUuWLNGAAQP0zjvv6OWXX1ZERIS6du2qp556SoMHD/aPmzdvnlq1aqUlS5Zoy5YtSk9P16effqqHH374mtvIzs7Wli1bNHfuXP32t7+Vz+dT9+7dNXnyZP+YsWPHasaMGVq5cqXee+89GWMaDKDevXvr888/V35+vgoKCuTz+ZSenq733nuvzmeAQqWkpES7du1SXl5enWtrQGO5TDCuTAIA4BC/ygAArCCAAABWEEAAACsIIACAFQQQAMAKAggAYEWT+xyQz+fT0aNHFR0d3ainBwMA7DLG6OTJk0pJSbnq58aaXAAdPXr0pj3bCgAQOocPH77qt+Y2uQCKjo6WJA3RSEUo0nI3AACnLui8tmm9///zhoQsgBYtWqQ33nhDFRUV6t+/v95++20NHDjwmnWX33aLUKQiXAQQADQ7///5Ote6jBKSmxA++OAD5eXlac6cOfryyy/Vv39/ZWdn+7/wCwCAkATQggULNHnyZE2aNEl33XWXlixZojZt2uhPf/pTKDYHAGiGgh5A586d065duwK+jCssLEyZmZkqLi6uM762tlZerzdgAgC0fEEPoO+++04XL15UYmJiwPzExERVVFTUGV9QUCCPx+OfuAMOAG4N1j+Imp+fr+rqav90+PBh2y0BAG6CoN8FFxcXp/DwcFVWVgbMr6ysVFJSUp3xbreb75cHgFtQ0M+AoqKiNGDAAG3evNk/z+fzafPmzRo0aFCwNwcAaKZC8jmgvLw8TZgwQffee68GDhyohQsXqqamRpMmTQrF5gAAzVBIAuixxx7T8ePHNXv2bFVUVOjuu+/Whg0b6tyYAAC4dbmMMcZ2Ez/k9Xrl8XiUodE8CQEAmqEL5rwKtVbV1dWKiYlpcJz1u+AAALcmAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACuCHkCvvfaaXC5XwNSrV69gbwYA0MxFhGKlvXv31qZNm/69kYiQbAYA0IyFJBkiIiKUlJQUilUDAFqIkFwDOnDggFJSUtStWzc9+eSTOnToUINja2tr5fV6AyYAQMsX9ABKT0/XsmXLtGHDBi1evFhlZWUaOnSoTp48We/4goICeTwe/5SamhrslgAATZDLGGNCuYGqqip16dJFCxYs0NNPP11neW1trWpra/2vvV6vUlNTlaHRinBFhrI1AEAIXDDnVai1qq6uVkxMTIPjQn53QPv27dWjRw8dPHiw3uVut1tutzvUbQAAmpiQfw7o1KlTKi0tVXJycqg3BQBoRoIeQC+88IKKior0zTff6B//+IceffRRhYeH6/HHHw/2pgAAzVjQ34I7cuSIHn/8cZ04cULx8fEaMmSItm/frvj4+GBvCgDQjAU9gFauXBnsVQIAWiCeBQcAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVoT8C+lwc52YPMhxTeef1v9lgdey/1ii45pztc6/5fa2953XtDlyynGNJPn2/KtRdQCc4wwIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVvA07BbmxZ+vcFwzru33jdtY98aVOZbhvOSbC6cbtak3jz/QqDrcPP881sVxTdvfehq1rYjNuxpVh+vDGRAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWMHDSFuYt14e77hmdr/G/R7S4WvjuOb7O12Oa6L6VTmumd/nI8c1kvS75B2Oa/52up3jmofbnHJcczOdMecc1+yobeu4JqPVecc1asTf0e2PTXW+HUk9NjeqDNeJMyAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIKHkbYwbf/T+YMa2/5nCBppQMxN2s7bSRmNqps3uKvjmpiig45r5mfc7rjmZoo443Nc03ZvueOajls/dFzTNyrScU2bb5zXIPQ4AwIAWEEAAQCscBxAW7du1ahRo5SSkiKXy6U1a9YELDfGaPbs2UpOTlbr1q2VmZmpAwcOBKtfAEAL4TiAampq1L9/fy1atKje5fPnz9dbb72lJUuWaMeOHWrbtq2ys7N19uzZG24WANByOL4JIScnRzk5OfUuM8Zo4cKFeuWVVzR69GhJ0rvvvqvExEStWbNG48c7/7ZOAEDLFNRrQGVlZaqoqFBmZqZ/nsfjUXp6uoqLi+utqa2tldfrDZgAAC1fUAOooqJCkpSYmBgwPzEx0b/sSgUFBfJ4PP4pNTU1mC0BAJoo63fB5efnq7q62j8dPnzYdksAgJsgqAGUlJQkSaqsrAyYX1lZ6V92JbfbrZiYmIAJANDyBTWA0tLSlJSUpM2bN/vneb1e7dixQ4MGDQrmpgAAzZzju+BOnTqlgwf//eiRsrIy7dmzR7GxsercubNmzpypefPm6Y477lBaWppeffVVpaSkaMyYMcHsGwDQzDkOoJ07d+qBBx7wv87Ly5MkTZgwQcuWLdOLL76ompoaTZkyRVVVVRoyZIg2bNigVq1aBa9rAECz5zLGGNtN/JDX65XH41GGRivCxQMEgebixM+cv81ePPf3jmsW/N9ejmu2ZnV3XCNJF8rrv3sXV3fBnFeh1qq6uvqq1/Wt3wUHALg1EUAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYIXjr2MA0PJFdEl1XPP7l50/2TrSFe64ZtWbmY5rOpYXO65B6HEGBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABW8DBSAHXsn3Wb45r73C7HNV+dO+O4JvZfpx3XoGniDAgAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArOBhpEALVvvwfY2q+/LHv2tEldtxxbTnn3dc0/of/3Rcg6aJMyAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIKHkQIt2KGcxv2O2c7l/MGij5c95LimzYb/dlxjHFegqeIMCABgBQEEALDCcQBt3bpVo0aNUkpKilwul9asWROwfOLEiXK5XAHTiBEjgtUvAKCFcBxANTU16t+/vxYtWtTgmBEjRqi8vNw/vf/++zfUJACg5XF8E0JOTo5ycnKuOsbtdispKanRTQEAWr6QXAMqLCxUQkKCevbsqWnTpunEiRMNjq2trZXX6w2YAAAtX9ADaMSIEXr33Xe1efNm/frXv1ZRUZFycnJ08eLFescXFBTI4/H4p9TU1GC3BABogoL+OaDx48f7/9y3b1/169dP3bt3V2FhoYYPH15nfH5+vvLy8vyvvV4vIQQAt4CQ34bdrVs3xcXF6eDBg/Uud7vdiomJCZgAAC1fyAPoyJEjOnHihJKTk0O9KQBAM+L4LbhTp04FnM2UlZVpz549io2NVWxsrObOnatx48YpKSlJpaWlevHFF3X77bcrOzs7qI0DAJo3xwG0c+dOPfDAA/7Xl6/fTJgwQYsXL9bevXv15z//WVVVVUpJSVFWVpZ++ctfyu12/mwpAEDL5TiAMjIyZEzDjwP85JNPbqghAPULi452XPPTodsatS2v76zjmmP/q5vjGnftfzmuQcvBs+AAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgRdC/khtAaBx4rbfjmnVxf2jUtkYfGOe4xr2eJ1vDGc6AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKHkYKWFD91I8c1+x97C3HNaUXzjuukaRTv+7kuMat8kZtC7cuzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoeRgrcoIjbUhzXzHz1A8c1bpfzf67j//unjmskKf7v/9WoOsAJzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoeRgr8gCvC+T+J/uuOOK75n+1OOK5ZfjLBcU3iq437HdPXqCrAGc6AAABWEEAAACscBVBBQYHuu+8+RUdHKyEhQWPGjFFJSUnAmLNnzyo3N1cdO3ZUu3btNG7cOFVWVga1aQBA8+cogIqKipSbm6vt27dr48aNOn/+vLKyslRTU+MfM2vWLH388cdatWqVioqKdPToUY0dOzbojQMAmjdHV1w3bNgQ8HrZsmVKSEjQrl27NGzYMFVXV+uPf/yjVqxYoQcffFCStHTpUt15553avn27fvSjHwWvcwBAs3ZD14Cqq6slSbGxsZKkXbt26fz588rMzPSP6dWrlzp37qzi4uJ611FbWyuv1xswAQBavkYHkM/n08yZMzV48GD16dNHklRRUaGoqCi1b98+YGxiYqIqKirqXU9BQYE8Ho9/Sk1NbWxLAIBmpNEBlJubq3379mnlypU31EB+fr6qq6v90+HDh29ofQCA5qFRH0SdPn261q1bp61bt6pTp07++UlJSTp37pyqqqoCzoIqKyuVlJRU77rcbrfcbndj2gAANGOOzoCMMZo+fbpWr16tzz77TGlpaQHLBwwYoMjISG3evNk/r6SkRIcOHdKgQYOC0zEAoEVwdAaUm5urFStWaO3atYqOjvZf1/F4PGrdurU8Ho+efvpp5eXlKTY2VjExMZoxY4YGDRrEHXAAgACOAmjx4sWSpIyMjID5S5cu1cSJEyVJv/vd7xQWFqZx48aptrZW2dnZ+sMf/hCUZgEALYfLGGNsN/FDXq9XHo9HGRqtCFek7XZwi3EN6O245m//+y8h6KSu/5Gf67im/bv1f/wBCKUL5rwKtVbV1dWKiYlpcBzPggMAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVjfpGVKCpC7+rR6PqpqxcG+RO6nfXn5w/2brrX7aHoBPAHs6AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKHkaKFmn/sx0aVTeqjTfIndSvU+E550XGBL8RwCLOgAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACh5Giibv7KiBjms2j/ptI7fWppF1AJziDAgAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArOBhpGjyjg4Od1zTOeLmPVR0+ckExzWR3nOOa4zjCqBp4wwIAGAFAQQAsMJRABUUFOi+++5TdHS0EhISNGbMGJWUlASMycjIkMvlCpieeeaZoDYNAGj+HAVQUVGRcnNztX37dm3cuFHnz59XVlaWampqAsZNnjxZ5eXl/mn+/PlBbRoA0Pw5uglhw4YNAa+XLVumhIQE7dq1S8OGDfPPb9OmjZKSkoLTIQCgRbqha0DV1dWSpNjY2ID5y5cvV1xcnPr06aP8/HydPn26wXXU1tbK6/UGTACAlq/Rt2H7fD7NnDlTgwcPVp8+ffzzn3jiCXXp0kUpKSnau3evXnrpJZWUlOijjz6qdz0FBQWaO3duY9sAADRTjQ6g3Nxc7du3T9u2bQuYP2XKFP+f+/btq+TkZA0fPlylpaXq3r17nfXk5+crLy/P/9rr9So1NbWxbQEAmolGBdD06dO1bt06bd26VZ06dbrq2PT0dEnSwYMH6w0gt9stt9vdmDYAAM2YowAyxmjGjBlavXq1CgsLlZaWds2aPXv2SJKSk5Mb1SAAoGVyFEC5ublasWKF1q5dq+joaFVUVEiSPB6PWrdurdLSUq1YsUIjR45Ux44dtXfvXs2aNUvDhg1Tv379QvIDAACaJ0cBtHjxYkmXPmz6Q0uXLtXEiRMVFRWlTZs2aeHChaqpqVFqaqrGjRunV155JWgNAwBaBsdvwV1NamqqioqKbqghAMCtgadhAz9QcOIuxzXF2V0d15jy/+O4BmhpeBgpAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFjBw0jR5HX7RbHjmpG/uCcEnTSk4iZuC2g5OAMCAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWNLlnwRljJEkXdF4ylpsBADh2Qecl/fv/84Y0uQA6efKkJGmb1lvuBABwI06ePCmPx9Pgcpe5VkTdZD6fT0ePHlV0dLRcLlfAMq/Xq9TUVB0+fFgxMTGWOrSP/XAJ++ES9sMl7IdLmsJ+MMbo5MmTSklJUVhYw1d6mtwZUFhYmDp16nTVMTExMbf0AXYZ++ES9sMl7IdL2A+X2N4PVzvzuYybEAAAVhBAAAArmlUAud1uzZkzR26323YrVrEfLmE/XMJ+uIT9cElz2g9N7iYEAMCtoVmdAQEAWg4CCABgBQEEALCCAAIAWEEAAQCsaDYBtGjRInXt2lWtWrVSenq6/vnPf9pu6aZ77bXX5HK5AqZevXrZbivktm7dqlGjRiklJUUul0tr1qwJWG6M0ezZs5WcnKzWrVsrMzNTBw4csNNsCF1rP0ycOLHO8TFixAg7zYZIQUGB7rvvPkVHRyshIUFjxoxRSUlJwJizZ88qNzdXHTt2VLt27TRu3DhVVlZa6jg0rmc/ZGRk1DkennnmGUsd169ZBNAHH3ygvLw8zZkzR19++aX69++v7OxsHTt2zHZrN13v3r1VXl7un7Zt22a7pZCrqalR//79tWjRonqXz58/X2+99ZaWLFmiHTt2qG3btsrOztbZs2dvcqehda39IEkjRowIOD7ef//9m9hh6BUVFSk3N1fbt2/Xxo0bdf78eWVlZammpsY/ZtasWfr444+1atUqFRUV6ejRoxo7dqzFroPvevaDJE2ePDngeJg/f76ljhtgmoGBAwea3Nxc/+uLFy+alJQUU1BQYLGrm2/OnDmmf//+ttuwSpJZvXq1/7XP5zNJSUnmjTfe8M+rqqoybrfbvP/++xY6vDmu3A/GGDNhwgQzevRoK/3YcuzYMSPJFBUVGWMu/d1HRkaaVatW+cd8/fXXRpIpLi621WbIXbkfjDHm/vvvN88//7y9pq5Dkz8DOnfunHbt2qXMzEz/vLCwMGVmZqq4uNhiZ3YcOHBAKSkp6tatm5588kkdOnTIdktWlZWVqaKiIuD48Hg8Sk9PvyWPj8LCQiUkJKhnz56aNm2aTpw4YbulkKqurpYkxcbGSpJ27dql8+fPBxwPvXr1UufOnVv08XDlfrhs+fLliouLU58+fZSfn6/Tp0/baK9BTe5p2Ff67rvvdPHiRSUmJgbMT0xM1P79+y11ZUd6erqWLVumnj17qry8XHPnztXQoUO1b98+RUdH227PioqKCkmq9/i4vOxWMWLECI0dO1ZpaWkqLS3Vyy+/rJycHBUXFys8PNx2e0Hn8/k0c+ZMDR48WH369JF06XiIiopS+/btA8a25OOhvv0gSU888YS6dOmilJQU7d27Vy+99JJKSkr00UcfWew2UJMPIPxbTk6O/8/9+vVTenq6unTpor/+9a96+umnLXaGpmD8+PH+P/ft21f9+vVT9+7dVVhYqOHDh1vsLDRyc3O1b9++W+I66NU0tB+mTJni/3Pfvn2VnJys4cOHq7S0VN27d7/Zbdaryb8FFxcXp/Dw8Dp3sVRWViopKclSV01D+/bt1aNHDx08eNB2K9ZcPgY4Purq1q2b4uLiWuTxMX36dK1bt05btmwJ+P6wpKQknTt3TlVVVQHjW+rx0NB+qE96erokNanjockHUFRUlAYMGKDNmzf75/l8Pm3evFmDBg2y2Jl9p06dUmlpqZKTk223Yk1aWpqSkpICjg+v16sdO3bc8sfHkSNHdOLEiRZ1fBhjNH36dK1evVqfffaZ0tLSApYPGDBAkZGRAcdDSUmJDh061KKOh2vth/rs2bNHkprW8WD7LojrsXLlSuN2u82yZcvMv/71LzNlyhTTvn17U1FRYbu1m+o//uM/TGFhoSkrKzNffPGFyczMNHFxcebYsWO2WwupkydPmt27d5vdu3cbSWbBggVm9+7d5ttvvzXGGPOrX/3KtG/f3qxdu9bs3bvXjB492qSlpZkzZ85Y7jy4rrYfTp48aV544QVTXFxsysrKzKZNm8w999xj7rjjDnP27FnbrQfNtGnTjMfjMYWFhaa8vNw/nT592j/mmWeeMZ07dzafffaZ2blzpxk0aJAZNGiQxa6D71r74eDBg+b11183O3fuNGVlZWbt2rWmW7duZtiwYZY7D9QsAsgYY95++23TuXNnExUVZQYOHGi2b99uu6Wb7rHHHjPJyckmKirK3Hbbbeaxxx4zBw8etN1WyG3ZssVIqjNNmDDBGHPpVuxXX33VJCYmGrfbbYYPH25KSkrsNh0CV9sPp0+fNllZWSY+Pt5ERkaaLl26mMmTJ7e4X9Lq+/klmaVLl/rHnDlzxjz77LOmQ4cOpk2bNubRRx815eXl9poOgWvth0OHDplhw4aZ2NhY43a7ze23325+/vOfm+rqaruNX4HvAwIAWNHkrwEBAFomAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACw4v8BT9CqtO7TXiQAAAAASUVORK5CYII=", "text/plain": [ - "
" + "['checkpoint.model.keras']" ] }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "plt.figure()\n", - "plt.title(\"Prediction: {}\".format(np.argmax(prediction)))\n", - "plt.imshow(test_img.reshape(28,28))\n", - "plt.show()" - ] - }, - { - "cell_type": "markdown", - "id": "135b42b0", - "metadata": {}, - "source": [ - "### Save Model" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "33f11a35", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "0" - ] - }, - "execution_count": 9, + "execution_count": 8, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "subprocess.call(\"rm -rf mnist_model\".split())" + "os.listdir(checkpoint_dir)" ] }, { "cell_type": "code", - "execution_count": 10, - "id": "86f9376d", + "execution_count": 9, + "id": "50eeb6e5", "metadata": {}, "outputs": [ { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ - "WARNING:absl:Found untraced functions such as _update_step_xla while saving (showing 1 of 1). These functions will not be directly callable after loading.\n" + "INFO:tensorflow:Assets written to: mnist_model/assets\n" ] }, { - "name": "stdout", + "name": "stderr", "output_type": "stream", "text": [ "INFO:tensorflow:Assets written to: mnist_model/assets\n" ] }, { - "name": "stderr", + "name": "stdout", "output_type": "stream", "text": [ - "INFO:tensorflow:Assets written to: mnist_model/assets\n" + "Saved artifact at 'mnist_model'. The following endpoints are available:\n", + "\n", + "* Endpoint 'serve'\n", + " args_0 (POSITIONAL_ONLY): TensorSpec(shape=(None, 784), dtype=tf.float32, name='keras_tensor')\n", + "Output Type:\n", + " TensorSpec(shape=(None, 10), dtype=tf.float32, name=None)\n", + "Captures:\n", + " 139403584120848: TensorSpec(shape=(), dtype=tf.resource, name=None)\n", + " 139403240100240: TensorSpec(shape=(), dtype=tf.resource, name=None)\n", + " 139403240100048: TensorSpec(shape=(), dtype=tf.resource, name=None)\n", + " 139403240099856: TensorSpec(shape=(), dtype=tf.resource, name=None)\n" ] } ], "source": [ - "model.save('mnist_model')" - ] - }, - { - "cell_type": "markdown", - "id": "43a417c6", - "metadata": {}, - "source": [ - "### Inspect saved model" + "# Export model in saved_model format\n", + "model.export(\"mnist_model\")" ] }, { "cell_type": "code", - "execution_count": 11, - "id": "53a8ba4a", + "execution_count": 10, + "id": "6d3bba9e", "metadata": {}, "outputs": [ { - "name": "stdout", + "name": "stderr", "output_type": "stream", "text": [ - "\u001b[01;34mmnist_model\u001b[00m\n", - "├── \u001b[01;34massets\u001b[00m\n", - "├── fingerprint.pb\n", - "├── keras_metadata.pb\n", - "├── saved_model.pb\n", - "└── \u001b[01;34mvariables\u001b[00m\n", - " ├── variables.data-00000-of-00001\n", - " └── variables.index\n", - "\n", - "2 directories, 5 files\n" + "/home/rishic/anaconda3/envs/spark-dl-tf/lib/python3.11/site-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n", + " super().__init__(activity_regularizer=activity_regularizer, **kwargs)\n" ] }, { - "data": { - "text/plain": [ - "0" - ] - }, - "execution_count": 11, - "metadata": {}, - "output_type": "execute_result" + "name": "stdout", + "output_type": "stream", + "text": [ + "32/32 - 0s - 10ms/step - loss: 2.4196 - sparse_categorical_accuracy: 0.0590\n", + "Untrained model, accuracy: 5.90%\n" + ] } ], "source": [ - "subprocess.call(\"tree mnist_model\".split())" + "# Create a basic model instance\n", + "model = create_model()\n", + "\n", + "# Evaluate the model\n", + "loss, acc = model.evaluate(test_images, test_labels, verbose=2)\n", + "print(\"Untrained model, accuracy: {:5.2f}%\".format(100 * acc))" ] }, { "cell_type": "code", - "execution_count": 12, - "id": "677e377a", + "execution_count": 11, + "id": "22ad1708", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "The given SavedModel SignatureDef contains the following input(s):\n", - " inputs['dense_input'] tensor_info:\n", - " dtype: DT_FLOAT\n", - " shape: (-1, 784)\n", - " name: serving_default_dense_input:0\n", - "The given SavedModel SignatureDef contains the following output(s):\n", - " outputs['dense_1'] tensor_info:\n", - " dtype: DT_FLOAT\n", - " shape: (-1, 10)\n", - " name: StatefulPartitionedCall:0\n", - "Method name is: tensorflow/serving/predict\n" + "32/32 - 0s - 713us/step - loss: 0.4145 - sparse_categorical_accuracy: 0.8780\n", + "Restored model, accuracy: 87.80%\n" ] }, { - "data": { - "text/plain": [ - "0" - ] - }, - "execution_count": 12, - "metadata": {}, - "output_type": "execute_result" + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/rishic/anaconda3/envs/spark-dl-tf/lib/python3.11/site-packages/keras/src/saving/saving_lib.py:713: UserWarning: Skipping variable loading for optimizer 'adam', because it has 2 variables whereas the saved optimizer has 10 variables. \n", + " saveable.load_own_variables(weights_store.get(inner_path))\n" + ] } ], "source": [ - "subprocess.call(\"saved_model_cli show --dir mnist_model --tag_set serve --signature_def serving_default\".split())" + "# Load the weights from the checkpoint\n", + "model.load_weights(checkpoint_path)\n", + "\n", + "# Re-evaluate the model\n", + "loss, acc = model.evaluate(test_images, test_labels, verbose=2)\n", + "print(\"Restored model, accuracy: {:5.2f}%\".format(100 * acc))" ] }, { "cell_type": "markdown", - "id": "6f013a4b", + "id": "1c097d63", "metadata": {}, "source": [ - "### Load model" + "### Checkpoint callback options" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "cb336e89", + "metadata": {}, + "outputs": [], + "source": [ + "!rm -rf training_2\n", + "!mkdir training_2" ] }, { "cell_type": "code", "execution_count": 13, - "id": "c41008f2", + "id": "750b6deb", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "Model: \"sequential\"\n", - "_________________________________________________________________\n", - " Layer (type) Output Shape Param # \n", - "=================================================================\n", - " dense (Dense) (None, 512) 401920 \n", - " \n", - " dropout (Dropout) (None, 512) 0 \n", - " \n", - " dense_1 (Dense) (None, 10) 5130 \n", - " \n", - "=================================================================\n", - "Total params: 407,050\n", - "Trainable params: 407,050\n", - "Non-trainable params: 0\n", - "_________________________________________________________________\n" + "\n", + "Epoch 5: saving model to training_2/cp-0005.weights.h5\n", + "\n", + "Epoch 10: saving model to training_2/cp-0010.weights.h5\n", + "\n", + "Epoch 15: saving model to training_2/cp-0015.weights.h5\n", + "\n", + "Epoch 20: saving model to training_2/cp-0020.weights.h5\n", + "\n", + "Epoch 25: saving model to training_2/cp-0025.weights.h5\n", + "\n", + "Epoch 30: saving model to training_2/cp-0030.weights.h5\n", + "\n", + "Epoch 35: saving model to training_2/cp-0035.weights.h5\n", + "\n", + "Epoch 40: saving model to training_2/cp-0040.weights.h5\n", + "\n", + "Epoch 45: saving model to training_2/cp-0045.weights.h5\n", + "\n", + "Epoch 50: saving model to training_2/cp-0050.weights.h5\n" ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "new_model = tf.keras.models.load_model('mnist_model')\n", - "new_model.summary()" + "# Include the epoch in the file name (uses `str.format`)\n", + "checkpoint_path = \"training_2/cp-{epoch:04d}.weights.h5\"\n", + "checkpoint_dir = os.path.dirname(checkpoint_path)\n", + "\n", + "batch_size = 32\n", + "\n", + "# Calculate the number of batches per epoch\n", + "import math\n", + "n_batches = len(train_images) / batch_size\n", + "n_batches = math.ceil(n_batches) # round up the number of batches to the nearest whole integer\n", + "\n", + "# Create a callback that saves the model's weights every 5 epochs\n", + "cp_callback = tf.keras.callbacks.ModelCheckpoint(\n", + " filepath=checkpoint_path, \n", + " verbose=1, \n", + " save_weights_only=True,\n", + " save_freq=5*n_batches)\n", + "\n", + "# Create a new model instance\n", + "model = create_model()\n", + "\n", + "# Save the weights using the `checkpoint_path` format\n", + "model.save_weights(checkpoint_path.format(epoch=0))\n", + "\n", + "# Train the model with the new callback\n", + "model.fit(train_images, \n", + " train_labels,\n", + " epochs=50, \n", + " batch_size=batch_size, \n", + " callbacks=[cp_callback],\n", + " validation_data=(test_images, test_labels),\n", + " verbose=0)" ] }, { "cell_type": "code", "execution_count": 14, - "id": "256662e8-30f1-437f-88b7-4534a7e76907", + "id": "1c43fd3d", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "[]" + "['cp-0000.weights.h5',\n", + " 'cp-0015.weights.h5',\n", + " 'cp-0010.weights.h5',\n", + " 'cp-0035.weights.h5',\n", + " 'cp-0020.weights.h5',\n", + " 'cp-0040.weights.h5',\n", + " 'cp-0050.weights.h5',\n", + " 'cp-0005.weights.h5',\n", + " 'cp-0045.weights.h5',\n", + " 'cp-0025.weights.h5',\n", + " 'cp-0030.weights.h5']" ] }, "execution_count": 14, @@ -465,45 +669,44 @@ } ], "source": [ - "new_model.inputs" + "os.listdir(checkpoint_dir)" ] }, { - "cell_type": "markdown", - "id": "351c7b84", + "cell_type": "code", + "execution_count": 15, + "id": "0d7ae715", "metadata": {}, + "outputs": [], "source": [ - "### Predict" + "latest = \"training_2/cp-0030.weights.h5\"" ] }, { "cell_type": "code", - "execution_count": 15, - "id": "5dbeeb98", + "execution_count": 16, + "id": "d345c6f7", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "1/1 [==============================] - 0s 30ms/step\n" + "32/32 - 0s - 9ms/step - loss: 0.4501 - sparse_categorical_accuracy: 0.8720\n", + "Restored model, accuracy: 87.20%\n" ] - }, - { - "data": { - "text/plain": [ - "array([[ -9.547884 , -4.7447677, -0.956994 , 1.0100659, -15.265252 ,\n", - " -7.811299 , -15.518889 , 14.113535 , -4.9397273, -4.3322315]],\n", - " dtype=float32)" - ] - }, - "execution_count": 15, - "metadata": {}, - "output_type": "execute_result" } ], "source": [ - "new_model.predict(test_images[:1])" + "# Create a new model instance\n", + "model = create_model()\n", + "\n", + "# Load the previously saved weights\n", + "model.load_weights(latest)\n", + "\n", + "# Re-evaluate the model from the latest checkpoint\n", + "loss, acc = model.evaluate(test_images, test_labels, verbose=2)\n", + "print(\"Restored model, accuracy: {:5.2f}%\".format(100 * acc))" ] }, { @@ -516,12 +719,44 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 17, "id": "7fcf07bb", "metadata": {}, "outputs": [], "source": [ - "import pandas as pd" + "import pandas as pd\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2c022c24", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" ] }, { @@ -534,17 +769,17 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 19, "id": "49ff5203", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "(10000, 784)" + "(1000, 784)" ] }, - "execution_count": 17, + "execution_count": 19, "metadata": {}, "output_type": "execute_result" } @@ -557,7 +792,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 20, "id": "182ee0c7", "metadata": {}, "outputs": [ @@ -565,14 +800,14 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 1min, sys: 0 ns, total: 1min\n", - "Wall time: 1min 2s\n" + "CPU times: user 134 ms, sys: 15.5 ms, total: 149 ms\n", + "Wall time: 1.36 s\n" ] } ], "source": [ "%%time\n", - "df = spark.createDataFrame(test_pdf)" + "df = spark.createDataFrame(test_pdf).repartition(8)" ] }, { @@ -585,7 +820,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 21, "id": "0061c39a-0871-429e-a4ff-751d26bf4b04", "metadata": {}, "outputs": [ @@ -593,8 +828,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "23/05/19 17:46:15 WARN package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.\n", - "23/05/19 17:46:15 WARN TaskSetManager: Stage 0 contains a task of very large size (7067 KiB). The maximum recommended task size is 1000 KiB.\n", + "24/10/03 17:40:32 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.\n", "[Stage 0:> (0 + 8) / 8]\r" ] }, @@ -602,8 +836,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 12.1 ms, sys: 0 ns, total: 12.1 ms\n", - "Wall time: 5.72 s\n" + "CPU times: user 2.49 ms, sys: 1.65 ms, total: 4.13 ms\n", + "Wall time: 1.66 s\n" ] }, { @@ -629,7 +863,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 22, "id": "302c73ec", "metadata": {}, "outputs": [ @@ -637,17 +871,17 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 261 ms, sys: 0 ns, total: 261 ms\n", - "Wall time: 258 ms\n" + "CPU times: user 6.71 ms, sys: 4.92 ms, total: 11.6 ms\n", + "Wall time: 11.4 ms\n" ] }, { "data": { "text/plain": [ - "(10000, 1)" + "(1000, 1)" ] }, - "execution_count": 20, + "execution_count": 22, "metadata": {}, "output_type": "execute_result" } @@ -661,7 +895,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 23, "id": "5495901b", "metadata": {}, "outputs": [ @@ -669,8 +903,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 12.5 s, sys: 0 ns, total: 12.5 s\n", - "Wall time: 12.6 s\n" + "CPU times: user 46.6 ms, sys: 4.71 ms, total: 51.3 ms\n", + "Wall time: 91.7 ms\n" ] } ], @@ -681,31 +915,16 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 24, "id": "5fa7faa8-c6bd-41b0-b5f7-fb121f0332e6", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "23/05/19 17:46:33 WARN TaskSetManager: Stage 1 contains a task of very large size (7070 KiB). The maximum recommended task size is 1000 KiB.\n", - "[Stage 1:> (0 + 8) / 8]\r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 9.88 ms, sys: 0 ns, total: 9.88 ms\n", - "Wall time: 1.36 s\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" + "CPU times: user 807 μs, sys: 724 μs, total: 1.53 ms\n", + "Wall time: 211 ms\n" ] } ], @@ -724,18 +943,10 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 25, "id": "3d4ca414", "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "23/05/19 17:46:34 WARN TaskSetManager: Stage 2 contains a task of very large size (7070 KiB). The maximum recommended task size is 1000 KiB.\n" - ] - } - ], + "outputs": [], "source": [ "spark.conf.set(\"spark.sql.execution.arrow.maxRecordsPerBatch\", \"128\")\n", "# This line will fail if the vectorized reader runs out of memory\n", @@ -760,7 +971,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 26, "id": "db30fba6-24d0-4c00-8502-04f9b10e7e16", "metadata": {}, "outputs": [], @@ -776,24 +987,34 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 27, "id": "b9cf62f8-96b2-4716-80bd-bb93d5f939bd", "metadata": {}, "outputs": [], "source": [ "# get absolute path to model\n", - "model_dir = \"{}/mnist_model\".format(os.getcwd())" + "model_dir = \"{}/training_1/checkpoint.model.keras\".format(os.getcwd())" ] }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 28, "id": "b81fa297-d9d0-4600-880d-dbdcdf8bccc6", "metadata": {}, "outputs": [], "source": [ "def predict_batch_fn():\n", " import tensorflow as tf\n", + "\n", + " # Enable GPU memory growth to avoid CUDA OOM\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + "\n", " model = tf.keras.models.load_model(model_dir)\n", " def predict(inputs: np.ndarray) -> np.ndarray:\n", " return model.predict(inputs)\n", @@ -803,7 +1024,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 29, "id": "72a689bd-dd82-492e-8740-1738a215325f", "metadata": {}, "outputs": [], @@ -816,7 +1037,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 30, "id": "60a70150-26b1-4145-9e7d-6e17389216b7", "metadata": {}, "outputs": [ @@ -826,7 +1047,7 @@ "1" ] }, - "execution_count": 28, + "execution_count": 30, "metadata": {}, "output_type": "execute_result" } @@ -838,7 +1059,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 31, "id": "e027f0d2-0f65-47b7-a562-2f0965faceec", "metadata": {}, "outputs": [ @@ -866,7 +1087,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 32, "id": "f0c3fb2e-469e-47bc-b948-8f6b0d7f6513", "metadata": {}, "outputs": [ @@ -874,15 +1095,22 @@ "name": "stderr", "output_type": "stream", "text": [ - " \r" + "[Stage 4:===================================================> (7 + 1) / 8]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 827 ms, sys: 0 ns, total: 827 ms\n", - "Wall time: 6.2 s\n" + "CPU times: user 18.5 ms, sys: 13.3 ms, total: 31.8 ms\n", + "Wall time: 5.03 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" ] } ], @@ -894,23 +1122,16 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 33, "id": "cdfa229a-f4a9-4c11-a410-de4a21c02c82", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 628 ms, sys: 0 ns, total: 628 ms\n", - "Wall time: 2.18 s\n" + "CPU times: user 37.3 ms, sys: 12.4 ms, total: 49.8 ms\n", + "Wall time: 259 ms\n" ] } ], @@ -921,23 +1142,16 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 34, "id": "5586ce49-6f93-4343-9b66-0dbb64972179", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 612 ms, sys: 0 ns, total: 612 ms\n", - "Wall time: 2.05 s\n" + "CPU times: user 22.9 ms, sys: 5.96 ms, total: 28.8 ms\n", + "Wall time: 237 ms\n" ] } ], @@ -958,10 +1172,17 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 35, "id": "4f947dc0-6b18-4605-810b-e83250a161db", "metadata": {}, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, { "data": { "text/html": [ @@ -991,60 +1212,60 @@ " \n", " 0\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-9.66701889038086, -3.6094393730163574, -3.31...\n", + " [-5.88436, -3.1058547, 0.10873719, 12.67319, -...\n", " \n", " \n", " 1\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-16.348304748535156, 3.1133787631988525, 6.73...\n", + " [-3.273286, -8.362554, 1.8936121, -3.8881433, ...\n", " \n", " \n", " 2\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-21.153539657592773, -6.192655086517334, -6.6...\n", + " [-3.3856308, 0.6785604, 1.3146863, 0.9275978, ...\n", " \n", " \n", " 3\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-13.79001522064209, -16.20423126220703, -8.87...\n", + " [-2.7754688, -7.3659225, 11.768427, 1.3434286,...\n", " \n", " \n", " 4\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-18.723485946655273, -7.806203365325928, -8.6...\n", + " [-4.9426627, 4.0774136, -0.4529277, -0.9312789...\n", " \n", " \n", " 5\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [10.495272636413574, -10.992117881774902, -2.4...\n", + " [-5.226616, -3.1389174, 2.6100307, 3.695045, -...\n", " \n", " \n", " 6\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-9.437854766845703, -3.7454347610473633, -3.5...\n", + " [-4.3006196, 5.1169925, 0.5850615, -0.76248693...\n", " \n", " \n", " 7\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-12.314886093139648, -2.106672763824463, -0.7...\n", + " [-2.3985956, -1.4814724, -4.884057, -0.2391600...\n", " \n", " \n", " 8\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-22.142608642578125, -2.8917746543884277, -4....\n", + " [0.82160115, -2.8640625, -1.6951559, -4.489290...\n", " \n", " \n", " 9\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-12.939343452453613, 0.40699127316474915, 15....\n", + " [-1.2338604, -2.151981, -4.171742, 1.6106845, ...\n", " \n", " \n", "\n", "" ], "text/plain": [ - " data \n", - "0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \\\n", + " data \\\n", + "0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", "1 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", "2 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", "3 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", @@ -1056,19 +1277,19 @@ "9 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", "\n", " preds \n", - "0 [-9.66701889038086, -3.6094393730163574, -3.31... \n", - "1 [-16.348304748535156, 3.1133787631988525, 6.73... \n", - "2 [-21.153539657592773, -6.192655086517334, -6.6... \n", - "3 [-13.79001522064209, -16.20423126220703, -8.87... \n", - "4 [-18.723485946655273, -7.806203365325928, -8.6... \n", - "5 [10.495272636413574, -10.992117881774902, -2.4... \n", - "6 [-9.437854766845703, -3.7454347610473633, -3.5... \n", - "7 [-12.314886093139648, -2.106672763824463, -0.7... \n", - "8 [-22.142608642578125, -2.8917746543884277, -4.... \n", - "9 [-12.939343452453613, 0.40699127316474915, 15.... " + "0 [-5.88436, -3.1058547, 0.10873719, 12.67319, -... \n", + "1 [-3.273286, -8.362554, 1.8936121, -3.8881433, ... \n", + "2 [-3.3856308, 0.6785604, 1.3146863, 0.9275978, ... \n", + "3 [-2.7754688, -7.3659225, 11.768427, 1.3434286,... \n", + "4 [-4.9426627, 4.0774136, -0.4529277, -0.9312789... \n", + "5 [-5.226616, -3.1389174, 2.6100307, 3.695045, -... \n", + "6 [-4.3006196, 5.1169925, 0.5850615, -0.76248693... \n", + "7 [-2.3985956, -1.4814724, -4.884057, -0.2391600... \n", + "8 [0.82160115, -2.8640625, -1.6951559, -4.489290... \n", + "9 [-1.2338604, -2.151981, -4.171742, 1.6106845, ... " ] }, - "execution_count": 33, + "execution_count": 35, "metadata": {}, "output_type": "execute_result" } @@ -1080,26 +1301,19 @@ }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 36, "id": "de4964e0-d1f8-4753-afa1-a8f95ca3f151", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "[-9.66701889038086,\n", - " -3.6094393730163574,\n", - " -3.3102951049804688,\n", - " -0.32629871368408203,\n", - " -8.564104080200195,\n", - " -3.614396095275879,\n", - " -5.390956878662109,\n", - " -3.0429909229278564,\n", - " 9.717466354370117,\n", - " -5.417511463165283]" + "array([ -5.88436 , -3.1058547 , 0.10873719, 12.67319 ,\n", + " -5.143787 , 4.0859914 , -10.203137 , -1.4333997 ,\n", + " -3.3865087 , -3.8473575 ], dtype=float32)" ] }, - "execution_count": 34, + "execution_count": 36, "metadata": {}, "output_type": "execute_result" } @@ -1111,7 +1325,7 @@ }, { "cell_type": "code", - "execution_count": 35, + "execution_count": 37, "id": "44e9a874-e301-4b72-8df7-bf1c5133c287", "metadata": {}, "outputs": [], @@ -1122,7 +1336,7 @@ }, { "cell_type": "code", - "execution_count": 36, + "execution_count": 38, "id": "c60e5af4-fc1e-4575-a717-f304664235be", "metadata": {}, "outputs": [], @@ -1133,13 +1347,13 @@ }, { "cell_type": "code", - "execution_count": 37, + "execution_count": 39, "id": "eb45ecc9-d376-40c4-ad7b-2bd08ca5aaf6", "metadata": {}, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAk1klEQVR4nO3dfXRU9b3v8c/kaXhKBkKeJWBAHixIrBQihwdRUkJQFk/HIz7cBegBxIAgtXpiVUS5Ny2eemhpBM+qEm1BkNMiS0tRnhJEgVMQDherKeRECRcSkCszEEiA5Hf/4DJ1IAH3OMkvCe/XWnstZu/fd/aXzYYPe/aeX1zGGCMAABpZmO0GAADXJwIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIcODGG2/U5MmT/a8LCwvlcrlUWFgYsn24XC698MILIXs/oKkigNBsFBQUyOVy+ZdWrVqpR48emjlzpioqKmy358i6deuaVci88847uv3229W+fXt17NhRd9xxh/70pz/ZbgvNXITtBgCnXnzxRaWlpamqqkrbtm3TkiVLtG7dOu3fv19t2rRp1F6GDh2qs2fPKioqylHdunXrlJ+fX2cInT17VhERTeev5uLFi/X444/r7rvv1s9//nNVVVWpoKBA99xzj/7whz9o/PjxtltEM9V0znLgO8rOztaPfvQjSdI///M/q2PHjnrllVe0du1a3X///XXWVFZWqm3btiHvJSwsTK1atQrpe4b6/b6vxYsXq3///nrvvffkcrkkSQ8//LBuuOEGvfnmmwQQgsZHcGj27rrrLklSaWmpJGny5Mlq166dSkpKNGrUKEVHR+vBBx+UJNXW1mrRokXq3bu3WrVqpcTERE2fPl3ffPNNwHsaY7RgwQJ16tRJbdq00Z133qnPPvvsin3Xdw9o586dGjVqlDp06KC2bduqb9+++tWvfuXvLz8/X5ICPlK8pK57QHv27FF2drZiYmLUrl07DR8+XDt27AgYc+kjyo8//lhz585VfHy82rZtq3Hjxun48eMBY71er7744gt5vd5rHl+fz6eEhISAHi/10bp162vWA/XhCgjNXklJiSSpY8eO/nUXLlxQVlaWBg8erH/913/1fzQ3ffp0FRQUaMqUKXr88cdVWlqq3/zmN9qzZ48+/vhjRUZGSpKef/55LViwQKNGjdKoUaP06aefasSIETp37tw1+9mwYYPuueceJScna/bs2UpKStLnn3+u999/X7Nnz9b06dN15MgRbdiwQb/73e+u+X6fffaZhgwZopiYGD311FOKjIzUa6+9pmHDhqmoqEgZGRkB42fNmqUOHTpo3rx5+vLLL7Vo0SLNnDlTq1at8o9Zs2aNpkyZomXLlgU8VFGXYcOG6T/+4z+0ePFijR49WlVVVVq8eLG8Xq9mz559zf6BehmgmVi2bJmRZDZu3GiOHz9uysrKzMqVK03Hjh1N69atzeHDh40xxkyaNMlIMv/yL/8SUP/RRx8ZSWb58uUB69evXx+w/tixYyYqKsrcfffdpra21j/umWeeMZLMpEmT/Ou2bNliJJktW7YYY4y5cOGCSUtLM126dDHffPNNwH6+/V45OTmmvr9+ksy8efP8r8eOHWuioqJMSUmJf92RI0dMdHS0GTp06BXHJzMzM2BfTzzxhAkPDzcnT568YuyyZcvq7OHbKioqzPDhw40k/xIXF2c++eSTa9YCV8NHcGh2MjMzFR8fr9TUVE2cOFHt2rXTmjVrdMMNNwSMmzFjRsDr1atXy+Px6Mc//rG+/vpr/9KvXz+1a9dOW7ZskSRt3LhR586d06xZswI+dpozZ841e9uzZ49KS0s1Z84ctW/fPmDbt9/ru6qpqdGHH36osWPHqmvXrv71ycnJeuCBB7Rt2zb5fL6AmmnTpgXsa8iQIaqpqdFXX33lXzd58mQZY6559SNJbdq0Uc+ePTVp0iStXr1ab7zxhpKTkzV+/HgdPHjQ8e8JuISP4NDs5Ofnq0ePHoqIiFBiYqJ69uypsLDA/0tFRESoU6dOAesOHDggr9erhISEOt/32LFjkuT/h7p79+4B2+Pj49WhQ4er9nbp48A+ffp899/QVRw/flxnzpxRz549r9h28803q7a2VmVlZerdu7d/fefOnQPGXer58vtc39W9996riIgIvffee/51Y8aMUffu3fWzn/0s4KM9wAkCCM3OgAED/E/B1cftdl8RSrW1tUpISNDy5cvrrImPjw9ZjzaFh4fXud4Y4/i9/vu//1vr16/Xv//7vwesj42N1eDBg/Xxxx8H1SMgEUC4jnTr1k0bN27UoEGDrvr0VpcuXSRdvGL69sdex48fv+ZVRLdu3SRJ+/fvV2ZmZr3jvuvHcfHx8WrTpo2Ki4uv2PbFF18oLCxMqamp3+m9gnHpC741NTVXbDt//rwuXLjQYPtGy8c9IFw3/umf/kk1NTV66aWXrth24cIFnTx5UtLFe0yRkZFavHhxwFXDokWLrrmP2267TWlpaVq0aJH//S759ntd+k7S5WMuFx4erhEjRmjt2rX68ssv/esrKiq0YsUKDR48WDExMdfs63Lf9THsm266SWFhYVq1alVA/4cPH9ZHH32kH/7wh473DVzCFRCuG3fccYemT5+uvLw87d27VyNGjFBkZKQOHDig1atX61e/+pX+8R//UfHx8XryySeVl5ene+65R6NGjdKePXv05z//WXFxcVfdR1hYmJYsWaLRo0fr1ltv1ZQpU5ScnKwvvvhCn332mT744ANJUr9+/SRJjz/+uLKyshQeHq6JEyfW+Z4LFizQhg0bNHjwYD322GOKiIjQa6+9purqai1cuDCoY/FdH8OOj4/Xww8/rN/+9rcaPny4xo8fr1OnTunVV1/V2bNnlZubG9T+AYkAwnVm6dKl6tevn1577TU988wzioiI0I033qiHHnpIgwYN8o9bsGCBWrVqpaVLl2rLli3KyMjQhx9+qLvvvvua+8jKytKWLVs0f/58/fKXv1Rtba26deumqVOn+seMHz9es2bN0sqVK/X73/9exph6A6h379766KOPlJubq7y8PNXW1iojI0O///3vr/gOUENYsmSJ0tPT9frrr/sDp3///nrrrbc0dOjQBt8/Wi6XCebOJAAA3xP3gAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsKLJfQ+otrZWR44cUXR0dFCzBwMA7DLG6NSpU0pJSbliTsZva3IBdOTIkQad2woA0DjKysqumJX+25pcAEVHR0uSBmuUIhRpuRsAgFMXdF7btM7/73l9GiyA8vPz9fLLL6u8vFzp6elavHixBgwYcM26Sx+7RShSES4CCACanf8/v861bqM0yEMIq1at0ty5czVv3jx9+umnSk9PV1ZWlv8HfgEA0CAB9Morr2jq1KmaMmWKfvCDH2jp0qVq06aN3njjjYbYHQCgGQp5AJ07d067d+8O+GFcYWFhyszM1Pbt268YX11dLZ/PF7AAAFq+kAfQ119/rZqaGiUmJgasT0xMVHl5+RXj8/Ly5PF4/AtPwAHA9cH6F1Fzc3Pl9Xr9S1lZme2WAACNIORPwcXFxSk8PNz/s+QvqaioUFJS0hXj3W633G53qNsAADRxIb8CioqKUr9+/bRp0yb/utraWm3atEkDBw4M9e4AAM1Ug3wPaO7cuZo0aZJ+9KMfacCAAVq0aJEqKys1ZcqUhtgdAKAZapAAuu+++3T8+HE9//zzKi8v16233qr169df8WACAOD65TLGGNtNfJvP55PH49EwjWEmBABohi6Y8yrUWnm9XsXExNQ7zvpTcACA6xMBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArIiw3QBwLeExMY5r/vZc76D29XDWZsc1z8QVO66pMbWOa768cMZxzagdjzmukaQui1yOa1zb/yuofeH6xRUQAMAKAggAYEXIA+iFF16Qy+UKWHr16hXq3QAAmrkGuQfUu3dvbdy48e87ieBWEwAgUIMkQ0REhJKSkhrirQEALUSD3AM6cOCAUlJS1LVrVz344IM6dOhQvWOrq6vl8/kCFgBAyxfyAMrIyFBBQYHWr1+vJUuWqLS0VEOGDNGpU6fqHJ+XlyePx+NfUlNTQ90SAKAJCnkAZWdn695771Xfvn2VlZWldevW6eTJk3rnnXfqHJ+bmyuv1+tfysrKQt0SAKAJavCnA9q3b68ePXro4MGDdW53u91yu90N3QYAoIlp8O8BnT59WiUlJUpOTm7oXQEAmpGQB9CTTz6poqIiffnll/rkk080btw4hYeH6/777w/1rgAAzVjIP4I7fPiw7r//fp04cULx8fEaPHiwduzYofj4+FDvCgDQjLmMMcZ2E9/m8/nk8Xg0TGMU4Yq03Q5CzAy61XHNi7973XFNf7fzyTTxd3+o7OC4piDjNsc1Nd9847gGTd8Fc16FWiuv16uYq0wmzFxwAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFg/9AOuDbXC9+7bgmmIlFV5/u6LhGkl7+248d10Suig1qX051ml73D3W8mlVdPwxqX2dqg/ghkTU1Qe0L1y+ugAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFs2GjUZ1eeoPzokXOS07WtHFeJCnxyVrHNTXFO4Lal1Nn/tTBcc2QrMeC2leU1/nM1m7fX4LaV2NwuYOY3VtSWLu2jmtqTvzfoPZ1PeIKCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsYDJSNCrPB587rll5Ot5xzVRPmeMaSTq0sqPjmt0/bJz/x9V8843jmuiVjTNRamNy9b/FcY33xTNB7Su+TaXjmpo7gtrVdYkrIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwgslI0ahqfD7HNb97aJTjmttWv+a4RpLmxm13XNP/tTmOa3pM/4vjmqYurFUrxzUl837ouGbnQ790XFMj47hGkob89qeOazqrPKh9XY+4AgIAWEEAAQCscBxAW7du1ejRo5WSkiKXy6V33303YLsxRs8//7ySk5PVunVrZWZm6sCBA6HqFwDQQjgOoMrKSqWnpys/P7/O7QsXLtSvf/1rLV26VDt37lTbtm2VlZWlqqqq790sAKDlcPwQQnZ2trKzs+vcZozRokWL9Oyzz2rMmDGSpLfeekuJiYl69913NXHixO/XLQCgxQjpPaDS0lKVl5crMzPTv87j8SgjI0Pbt9f9dFF1dbV8Pl/AAgBo+UIaQOXlFx8/TExMDFifmJjo33a5vLw8eTwe/5KamhrKlgAATZT1p+Byc3Pl9Xr9S1lZme2WAACNIKQBlJSUJEmqqKgIWF9RUeHfdjm3262YmJiABQDQ8oU0gNLS0pSUlKRNmzb51/l8Pu3cuVMDBw4M5a4AAM2c46fgTp8+rYMHD/pfl5aWau/evYqNjVXnzp01Z84cLViwQN27d1daWpqee+45paSkaOzYsaHsGwDQzDkOoF27dunOO+/0v547d64kadKkSSooKNBTTz2lyspKTZs2TSdPntTgwYO1fv16tQpinigAQMvlMsYEN0tfA/H5fPJ4PBqmMYpwRdpuB83V7X2DKlu+eonjmtog9pOxbo7jmmAmMA1mglBJKn/kNsc1907fdO1Bl3m64+eOa/JO/MBxTeHsf3BcI0nhWz4Nqu56d8GcV6HWyuv1XvW+vvWn4AAA1ycCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsYDZs4FvOjM9wXFO42PkM2idqzzquufPVnzquaTv4uOMaSdp+66qg6pzq88kkxzU3/o8Djmtqq6oc1yB4zIYNAGjSCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFhO0GgKak3br/clzTfdgMxzUHJjifwHTfzN84rglWMJOl/sM7P3Fc0/1nex3XMLFoy8EVEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYwWSkwLcYYxzXtPsqvAE6CY1gJhWVpJH/60nHNd2WbHdcU+u4Ai0JV0AAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAWTkaJFcrndQdUd/J8/dFxTfP9vgtpXY/jtN7cFVRcfxMSigFNcAQEArCCAAABWOA6grVu3avTo0UpJSZHL5dK7774bsH3y5MlyuVwBy8iRI0PVLwCghXAcQJWVlUpPT1d+fn69Y0aOHKmjR4/6l7fffvt7NQkAaHkcP4SQnZ2t7Ozsq45xu91KSkoKuikAQMvXIPeACgsLlZCQoJ49e2rGjBk6ceJEvWOrq6vl8/kCFgBAyxfyABo5cqTeeustbdq0Sb/4xS9UVFSk7Oxs1dTU1Dk+Ly9PHo/Hv6Smpoa6JQBAExTy7wFNnDjR/+tbbrlFffv2Vbdu3VRYWKjhw4dfMT43N1dz5871v/b5fIQQAFwHGvwx7K5duyouLk4HDx6sc7vb7VZMTEzAAgBo+Ro8gA4fPqwTJ04oOTm5oXcFAGhGHH8Ed/r06YCrmdLSUu3du1exsbGKjY3V/PnzNWHCBCUlJamkpERPPfWUbrrpJmVlZYW0cQBA8+Y4gHbt2qU777zT//rS/ZtJkyZpyZIl2rdvn958802dPHlSKSkpGjFihF566SW5g5ybCwDQMjkOoGHDhskYU+/2Dz744Hs1BITC8cnBTcJZfH/9X7AOpdWnOzquubdd/V9nqM+0Dp86rpGkbb0fdlxT81lxUPvC9Yu54AAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFyH8kNxBq4R06OK6Z9+SbDdBJ3fJO/MBxzdZHMxzX3Lt6meOaDmGtHddI0tlO0Y5roj4Lale4jnEFBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWMBkpmrzy+3o5rrm7zaag9rXprNtxzSc/7uy4JqzTBcc1jelsnPN/GqIaoA+0bFwBAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVTEaKRhUeE+O45qUnlzVAJ3XLKx3luCaq4ivHNa5O8Y5rgnHowpmg6mL/ctxxTU1Qe8L1jCsgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCyUjRqI4+1MdxzcjWhY5r/na+ynGNJLWe7XZcY6KjHdec+8UpxzXBeNvbL6i6mr+VhLgT4EpcAQEArCCAAABWOAqgvLw89e/fX9HR0UpISNDYsWNVXFwcMKaqqko5OTnq2LGj2rVrpwkTJqiioiKkTQMAmj9HAVRUVKScnBzt2LFDGzZs0Pnz5zVixAhVVlb6xzzxxBN67733tHr1ahUVFenIkSMaP358yBsHADRvjh5CWL9+fcDrgoICJSQkaPfu3Ro6dKi8Xq9ef/11rVixQnfddZckadmyZbr55pu1Y8cO3X777aHrHADQrH2ve0Ber1eSFBsbK0navXu3zp8/r8zMTP+YXr16qXPnztq+fXud71FdXS2fzxewAABavqADqLa2VnPmzNGgQYPUp8/FR2vLy8sVFRWl9u3bB4xNTExUeXl5ne+Tl5cnj8fjX1JTU4NtCQDQjAQdQDk5Odq/f79Wrlz5vRrIzc2V1+v1L2VlZd/r/QAAzUNQX0SdOXOm3n//fW3dulWdOnXyr09KStK5c+d08uTJgKugiooKJSUl1flebrdbbrfzL/8BAJo3R1dAxhjNnDlTa9as0ebNm5WWlhawvV+/foqMjNSmTZv864qLi3Xo0CENHDgwNB0DAFoER1dAOTk5WrFihdauXavo6Gj/fR2Px6PWrVvL4/HokUce0dy5cxUbG6uYmBjNmjVLAwcO5Ak4AEAARwG0ZMkSSdKwYcMC1i9btkyTJ0+WJP3bv/2bwsLCNGHCBFVXVysrK0uvvvpqSJoFALQcjgLIGHPNMa1atVJ+fr7y8/ODbgrNQ1jbto5rFsx9owE6udJKb/+g6mr++jfHNWXP/oPjmv99828c1xy6cMZxzZ9eGua4RpLaaWdQdYATzAUHALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAK4L6iaiAJJkLFxzX/LXqBsc1I1sfcFzzwf+52XGNJNVOindcs+vRRUHsyflfvbv+NNdxTY/VzGqNposrIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwgslIETRTXe24pmB5luOauTOdT0b6UfoqxzWSVJte67imouac45o7i6Y7rukxc7fjGqAp4woIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKxgMlI0qi5LP3dcM+++dMc18+P/y3GNJN3zxTjHNTUvJTiuuanwU8c1QEvDFRAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWMFkpGhUNd9847jmL7eGO64Zpdsc11x02HFFeBA1ALgCAgBYQgABAKxwFEB5eXnq37+/oqOjlZCQoLFjx6q4uDhgzLBhw+RyuQKWRx99NKRNAwCaP0cBVFRUpJycHO3YsUMbNmzQ+fPnNWLECFVWVgaMmzp1qo4ePepfFi5cGNKmAQDNn6OHENavXx/wuqCgQAkJCdq9e7eGDh3qX9+mTRslJSWFpkMAQIv0ve4Beb1eSVJsbGzA+uXLlysuLk59+vRRbm6uzpw5U+97VFdXy+fzBSwAgJYv6Mewa2trNWfOHA0aNEh9+vTxr3/ggQfUpUsXpaSkaN++fXr66adVXFysP/7xj3W+T15enubPnx9sGwCAZspljDHBFM6YMUN//vOftW3bNnXq1KnecZs3b9bw4cN18OBBdevW7Yrt1dXVqq6u9r/2+XxKTU3VMI1RhCsymNYAABZdMOdVqLXyer2KiYmpd1xQV0AzZ87U+++/r61bt141fCQpIyNDkuoNILfbLbfbHUwbAIBmzFEAGWM0a9YsrVmzRoWFhUpLS7tmzd69eyVJycnJQTUIAGiZHAVQTk6OVqxYobVr1yo6Olrl5eWSJI/Ho9atW6ukpEQrVqzQqFGj1LFjR+3bt09PPPGEhg4dqr59+zbIbwAA0Dw5ugfkcrnqXL9s2TJNnjxZZWVleuihh7R//35VVlYqNTVV48aN07PPPnvVzwG/zefzyePxcA8IAJqpBrkHdK2sSk1NVVFRkZO3BABcp5gLDgBgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgRYTtBi5njJEkXdB5yVhuBgDg2AWdl/T3f8/r0+QC6NSpU5KkbVpnuRMAwPdx6tQpeTyeere7zLUiqpHV1tbqyJEjio6OlsvlCtjm8/mUmpqqsrIyxcTEWOrQPo7DRRyHizgOF3EcLmoKx8EYo1OnTiklJUVhYfXf6WlyV0BhYWHq1KnTVcfExMRc1yfYJRyHizgOF3EcLuI4XGT7OFztyucSHkIAAFhBAAEArGhWAeR2uzVv3jy53W7brVjFcbiI43ARx+EijsNFzek4NLmHEAAA14dmdQUEAGg5CCAAgBUEEADACgIIAGAFAQQAsKLZBFB+fr5uvPFGtWrVShkZGfrP//xP2y01uhdeeEEulytg6dWrl+22GtzWrVs1evRopaSkyOVy6d133w3YbozR888/r+TkZLVu3VqZmZk6cOCAnWYb0LWOw+TJk684P0aOHGmn2QaSl5en/v37Kzo6WgkJCRo7dqyKi4sDxlRVVSknJ0cdO3ZUu3btNGHCBFVUVFjquGF8l+MwbNiwK86HRx991FLHdWsWAbRq1SrNnTtX8+bN06effqr09HRlZWXp2LFjtltrdL1799bRo0f9y7Zt22y31OAqKyuVnp6u/Pz8OrcvXLhQv/71r7V06VLt3LlTbdu2VVZWlqqqqhq504Z1reMgSSNHjgw4P95+++1G7LDhFRUVKScnRzt27NCGDRt0/vx5jRgxQpWVlf4xTzzxhN577z2tXr1aRUVFOnLkiMaPH2+x69D7LsdBkqZOnRpwPixcuNBSx/UwzcCAAQNMTk6O/3VNTY1JSUkxeXl5FrtqfPPmzTPp6em227BKklmzZo3/dW1trUlKSjIvv/yyf93JkyeN2+02b7/9toUOG8flx8EYYyZNmmTGjBljpR9bjh07ZiSZoqIiY8zFP/vIyEizevVq/5jPP//cSDLbt2+31WaDu/w4GGPMHXfcYWbPnm2vqe+gyV8BnTt3Trt371ZmZqZ/XVhYmDIzM7V9+3aLndlx4MABpaSkqGvXrnrwwQd16NAh2y1ZVVpaqvLy8oDzw+PxKCMj47o8PwoLC5WQkKCePXtqxowZOnHihO2WGpTX65UkxcbGSpJ2796t8+fPB5wPvXr1UufOnVv0+XD5cbhk+fLliouLU58+fZSbm6szZ87YaK9eTW427Mt9/fXXqqmpUWJiYsD6xMREffHFF5a6siMjI0MFBQXq2bOnjh49qvnz52vIkCHav3+/oqOjbbdnRXl5uSTVeX5c2na9GDlypMaPH6+0tDSVlJTomWeeUXZ2trZv367w8HDb7YVcbW2t5syZo0GDBqlPnz6SLp4PUVFRat++fcDYlnw+1HUcJOmBBx5Qly5dlJKSon379unpp59WcXGx/vjHP1rsNlCTDyD8XXZ2tv/Xffv2VUZGhrp06aJ33nlHjzzyiMXO0BRMnDjR/+tbbrlFffv2Vbdu3VRYWKjhw4db7Kxh5OTkaP/+/dfFfdCrqe84TJs2zf/rW265RcnJyRo+fLhKSkrUrVu3xm6zTk3+I7i4uDiFh4df8RRLRUWFkpKSLHXVNLRv3149evTQwYMHbbdizaVzgPPjSl27dlVcXFyLPD9mzpyp999/X1u2bAn4+WFJSUk6d+6cTp48GTC+pZ4P9R2HumRkZEhSkzofmnwARUVFqV+/ftq0aZN/XW1trTZt2qSBAwda7My+06dPq6SkRMnJybZbsSYtLU1JSUkB54fP59POnTuv+/Pj8OHDOnHiRIs6P4wxmjlzptasWaPNmzcrLS0tYHu/fv0UGRkZcD4UFxfr0KFDLep8uNZxqMvevXslqWmdD7afgvguVq5cadxutykoKDB//etfzbRp00z79u1NeXm57dYa1U9+8hNTWFhoSktLzccff2wyMzNNXFycOXbsmO3WGtSpU6fMnj17zJ49e4wk88orr5g9e/aYr776yhhjzM9//nPTvn17s3btWrNv3z4zZswYk5aWZs6ePWu589C62nE4deqUefLJJ8327dtNaWmp2bhxo7nttttM9+7dTVVVle3WQ2bGjBnG4/GYwsJCc/ToUf9y5swZ/5hHH33UdO7c2WzevNns2rXLDBw40AwcONBi16F3reNw8OBB8+KLL5pdu3aZ0tJSs3btWtO1a1czdOhQy50HahYBZIwxixcvNp07dzZRUVFmwIABZseOHbZbanT33XefSU5ONlFRUeaGG24w9913nzl48KDtthrcli1bjKQrlkmTJhljLj6K/dxzz5nExETjdrvN8OHDTXFxsd2mG8DVjsOZM2fMiBEjTHx8vImMjDRdunQxU6dObXH/Savr9y/JLFu2zD/m7Nmz5rHHHjMdOnQwbdq0MePGjTNHjx6113QDuNZxOHTokBk6dKiJjY01brfb3HTTTeanP/2p8Xq9dhu/DD8PCABgRZO/BwQAaJkIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCK/wcgFXDPfhuVLAAAAABJRU5ErkJggg==", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAkfElEQVR4nO3dfXQUdZ7v8U/nqSEkaR7yLAFCFHRAcAYly/AgSiQEZUCYGUG9F7gziJiggI6KR0Udzsksrg7qIHjcHVhHEGWOyMoiDg9JGBRwwTCIM2QhJ0g4kIBcSYcAIaR/9w+uvbQkQDUdfkl4v86pc+iq37fqm6Lgk+qqrnYZY4wAALjKwmw3AAC4NhFAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAgAPdunXTpEmT/K8LCwvlcrlUWFgYsm24XC698MILIVsf0FwRQGgxlixZIpfL5Z/atGmjHj16KC8vT5WVlbbbc2TNmjUtJmTefvtt3X777UpKSpLb7VZ6eromT56s/fv3224NLVyE7QYAp1566SWlp6fr9OnT2rx5sxYuXKg1a9Zo9+7dio6Ovqq9DBkyRKdOnVJUVJSjujVr1mjBggUNhtCpU6cUEdF8/mkWFxcrPT1dP/vZz9ShQweVlZXp7bff1urVq/W3v/1NqamptltEC9V8jnLgMuXk5OjWW2+VJP36179Wp06d9Oqrr2rVqlWaMGFCgzU1NTVq165dyHsJCwtTmzZtQrrOUK/vSr355psXzBszZoxuvfVWvfPOO3r66actdIXWgLfg0OLdeeedkqSysjJJ0qRJkxQTE6PS0lKNHDlSsbGxeuCBByRJPp9P8+fPV69evdSmTRslJSVp6tSp+u677wLWaYzR3Llz1blzZ0VHR+uOO+7Q119/fcG2G7sGtG3bNo0cOVIdOnRQu3bt1KdPH7322mv+/hYsWCBJAW8pfq+ha0DFxcXKyclRXFycYmJiNGzYMG3dujVgzPdvUX722WeaNWuWEhIS1K5dO9177706evRowNiqqirt2bNHVVVVl7OLL9CtWzdJ0vHjx4OqByTOgNAKlJaWSpI6derkn3f27FllZ2dr0KBB+pd/+Rf/W3NTp07VkiVLNHnyZD366KMqKyvTH/7wBxUXF+uzzz5TZGSkJOn555/X3LlzNXLkSI0cOVJffvmlhg8frjNnzlyyn3Xr1umee+5RSkqKHnvsMSUnJ+sf//iHVq9erccee0xTp07VoUOHtG7dOv3pT3+65Pq+/vprDR48WHFxcXryyScVGRmpt956S0OHDlVRUZEyMzMDxk+fPl0dOnTQnDlztH//fs2fP195eXl6//33/WNWrlypyZMna/HixQE3VVzMsWPHVF9frwMHDuill16SJA0bNuyyaoEGGaCFWLx4sZFk1q9fb44ePWrKy8vN8uXLTadOnUzbtm3NwYMHjTHGTJw40UgyTz/9dED9X//6VyPJLF26NGD+2rVrA+YfOXLEREVFmbvvvtv4fD7/uGeeecZIMhMnTvTPKygoMJJMQUGBMcaYs2fPmvT0dNO1a1fz3XffBWzn/HXl5uaaxv75STJz5szxvx4zZoyJiooypaWl/nmHDh0ysbGxZsiQIRfsn6ysrIBtzZw504SHh5vjx49fMHbx4sUN9tAQt9ttJBlJplOnTub111+/7FqgIbwFhxYnKytLCQkJSktL0/jx4xUTE6OVK1fquuuuCxg3bdq0gNcrVqyQx+PRXXfdpW+//dY/9evXTzExMSooKJAkrV+/XmfOnNH06dMD3hqbMWPGJXsrLi5WWVmZZsyYofbt2wcsO39dl6u+vl5/+ctfNGbMGHXv3t0/PyUlRffff782b94sr9cbUPPQQw8FbGvw4MGqr6/XN9984583adIkGWMu++xHkj755BOtWbNGr7zyirp06aKamhrHPw9wPt6CQ4uzYMEC9ejRQxEREUpKSlLPnj0VFhb4u1RERIQ6d+4cMG/v3r2qqqpSYmJig+s9cuSIJPn/o77hhhsClickJKhDhw4X7e37twN79+59+T/QRRw9elQnT55Uz549L1h20003yefzqby8XL169fLP79KlS8C473v+4XUup+644w5J524CGT16tHr37q2YmBjl5eVd0Xpx7SKA0OL079/ffxdcY9xu9wWh5PP5lJiYqKVLlzZYk5CQELIebQoPD29wvjEmZNvIyMjQj3/8Yy1dupQAQtAIIFwzMjIytH79eg0cOFBt27ZtdFzXrl0lnTtjOv9tr6NHj17yLCIjI0OStHv3bmVlZTU67nLfjktISFB0dLRKSkouWLZnzx6FhYUpLS3tstYVaqdOnVJtba2VbaN14BoQrhm//OUvVV9fr9/+9rcXLDt79qz/luKsrCxFRkbqjTfeCDhrmD9//iW38ZOf/ETp6emaP3/+Bbcon7+u7z+TdKnbmMPDwzV8+HCtWrUq4MkDlZWVWrZsmQYNGqS4uLhL9vVDl3sb9tmzZxsM3S+++EJfffXVJc9EgYvhDAjXjNtvv11Tp05Vfn6+du7cqeHDhysyMlJ79+7VihUr9Nprr+nnP/+5EhIS9MQTTyg/P1/33HOPRo4cqeLiYn3yySeKj4+/6DbCwsK0cOFCjRo1SrfccosmT56slJQU7dmzR19//bU+/fRTSVK/fv0kSY8++qiys7MVHh6u8ePHN7jOuXPnat26dRo0aJAeeeQRRURE6K233lJtba3mzZsX1L643NuwT5w4obS0NN13333q1auX2rVrp6+++kqLFy+Wx+PRc889F9T2AYkAwjVm0aJF6tevn9566y0988wzioiIULdu3fTggw9q4MCB/nFz585VmzZttGjRIhUUFCgzM1N/+ctfdPfdd19yG9nZ2SooKNCLL76oV155RT6fTxkZGZoyZYp/zNixYzV9+nQtX75c7777rowxjQZQr1699Ne//lWzZ89Wfn6+fD6fMjMz9e67717wGaBQi46O1q9//WsVFBToz3/+s06dOqXU1FRNmDBBzz77rP8DqUAwXCaUVyYBALhMXAMCAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCKZvc5IJ/Pp0OHDik2NjaopwcDAOwyxqi6ulqpqakXPJPxfM0ugA4dOmTt2VYAgNApLy+/4Kn052t2ARQbGytJGqSRilCk5W4AAE6dVZ02a43///PGNFkALViwQC+//LIqKirUt29fvfHGG+rfv/8l675/2y1CkYpwEUAA0OL8/+frXOoySpPchPD+++9r1qxZmjNnjr788kv17dtX2dnZ/i/8AgCgSQLo1Vdf1ZQpUzR58mT96Ec/0qJFixQdHa0//vGPTbE5AEALFPIAOnPmjHbs2BHwZVxhYWHKysrSli1bLhhfW1srr9cbMAEAWr+QB9C3336r+vp6JSUlBcxPSkpSRUXFBePz8/Pl8Xj8E3fAAcC1wfoHUWfPnq2qqir/VF5ebrslAMBVEPK74OLj4xUeHq7KysqA+ZWVlUpOTr5gvNvtltvtDnUbAIBmLuRnQFFRUerXr582bNjgn+fz+bRhwwYNGDAg1JsDALRQTfI5oFmzZmnixIm69dZb1b9/f82fP181NTWaPHlyU2wOANACNUkA3XfffTp69Kief/55VVRU6JZbbtHatWsvuDEBAHDtchljjO0mzuf1euXxeDRUo3kSAgC0QGdNnQq1SlVVVYqLi2t0nPW74AAA1yYCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWBFhuwHYZ37aN7i6cOe/v0RWeh3XlP7vRMc1vu6nHNdI0p7b/+i4JtzlfD9MPTjAcU3hp7c4run2nzWOayRJW3cFVwc4wBkQAMAKAggAYEXIA+iFF16Qy+UKmG688cZQbwYA0MI1yTWgXr16af369f+zkQguNQEAAjVJMkRERCg5ObkpVg0AaCWa5BrQ3r17lZqaqu7du+uBBx7QgQMHGh1bW1srr9cbMAEAWr+QB1BmZqaWLFmitWvXauHChSorK9PgwYNVXV3d4Pj8/Hx5PB7/lJaWFuqWAADNUMgDKCcnR7/4xS/Up08fZWdna82aNTp+/Lg++OCDBsfPnj1bVVVV/qm8vDzULQEAmqEmvzugffv26tGjh/bt29fgcrfbLbfb3dRtAACamSb/HNCJEydUWlqqlJSUpt4UAKAFCXkAPfHEEyoqKtL+/fv1+eef695771V4eLgmTJgQ6k0BAFqwkL8Fd/DgQU2YMEHHjh1TQkKCBg0apK1btyohISHUmwIAtGAuY4yx3cT5vF6vPB6Phmq0IlyRttuxqubnmY5rKm91flK7dsLLjmskqUtEW8c1/2v/XY5r/tRtneManFN8xhdU3eOP5zmuif5wW1DbQutz1tSpUKtUVVWluLi4RsfxLDgAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIKHkV4lR/J+6rim8OlXHNdEu6Ic1zR339afclzTxhXc71Z1cv7PYcaBexzX/DLxvxzX3B1d5bgmWPvqah3XPDH4l45rzpYfdFyD5o+HkQIAmjUCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsiLDdwLXCF+68pjU+2frlYz9yXLNhxiDHNfVtg/vd6rvrnT+B/br/POy45s2EcY5r7v7zHx3XBGvsf011XNPt+P7QN4JWjTMgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCh5FeJan/+jfHNR88kui4Jjv6gOOanDlPOK6RpLoYl+Oa6/7joOOaiP07nNc4rjgnOYia+iBqKu/5aRBVV8+uny5xXDMmiAes+qqrHdeg9eAMCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCs4GGkV4mvpsZxzTs90xzXvJ0z1nFNfEGx4xpJ8p0+7bjmbFBbunrCExIc13x3V4bjmscf/sBxDdDacAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcByY4yef/55paSkqG3btsrKytLevXtD1S8AoJVwHEA1NTXq27evFixY0ODyefPm6fXXX9eiRYu0bds2tWvXTtnZ2TodxPUCAEDr5fgmhJycHOXk5DS4zBij+fPn69lnn9Xo0aMlSe+8846SkpL00Ucfafz48VfWLQCg1QjpNaCysjJVVFQoKyvLP8/j8SgzM1NbtmxpsKa2tlZerzdgAgC0fiENoIqKCklSUlJSwPykpCT/sh/Kz8+Xx+PxT2lpzm89BgC0PNbvgps9e7aqqqr8U3l5ue2WAABXQUgDKDk5WZJUWVkZML+ystK/7Ifcbrfi4uICJgBA6xfSAEpPT1dycrI2bNjgn+f1erVt2zYNGDAglJsCALRwju+CO3HihPbt2+d/XVZWpp07d6pjx47q0qWLZsyYoblz5+qGG25Qenq6nnvuOaWmpmrMmDGh7BsA0MI5DqDt27frjjvu8L+eNWuWJGnixIlasmSJnnzySdXU1Oihhx7S8ePHNWjQIK1du1Zt2rQJXdcAgBbPZYwxtps4n9frlcfj0VCNVoQr0nY7aKHC23uCqnt8x2bHNUPanAlqW1eDT76g6n73bV/HNduGpTiuqf/2mOMaNH9nTZ0KtUpVVVUXva5v/S44AMC1iQACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACscfx0D0BKUPdorqLohbTaGuBO7VtXEB1X3ed+oIKp4sjWc4QwIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKzgYaRAK3ZX28NB1c2d9YDjmrqYoDblWGLxWcc1bT7+ogk6wZXiDAgAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArOBhpGiV0jacDKpuxyTnNf3cQW3qqogJC665HY+/EeJOQmfOkR87rtnxMb9rN0f8rQAArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFTyMFK2S67OdQdW9NOp+xzW1STGOa6of9zqu+eyW5Y5rWqNnE7Y7rrnzwUeD2pbn3a1B1eHycAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcDySZMmyeVyBUwjRowIVb8AgFbCcQDV1NSob9++WrBgQaNjRowYocOHD/un995774qaBAC0Po5vQsjJyVFOTs5Fx7jdbiUnJwfdFACg9WuSa0CFhYVKTExUz549NW3aNB07dqzRsbW1tfJ6vQETAKD1C3kAjRgxQu+88442bNigf/7nf1ZRUZFycnJUX1/f4Pj8/Hx5PB7/lJaWFuqWAADNUMg/BzR+/Hj/n2+++Wb16dNHGRkZKiws1LBhwy4YP3v2bM2aNcv/2uv1EkIAcA1o8tuwu3fvrvj4eO3bt6/B5W63W3FxcQETAKD1a/IAOnjwoI4dO6aUlJSm3hQAoAVx/BbciRMnAs5mysrKtHPnTnXs2FEdO3bUiy++qHHjxik5OVmlpaV68skndf311ys7OzukjQMAWjbHAbR9+3bdcccd/tffX7+ZOHGiFi5cqF27dunf//3fdfz4caWmpmr48OH67W9/K7fbHbquAQAtnuMAGjp0qIwxjS7/9NNPr6ghwKb6r0sc10R87Xw7HQpcjmtGRf3Ucc3+P/VwXCNJn2QudFzTOaJtUNtyKtIV7rjmdMfgrjZ4gqrC5eJZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAi5F/JDeAyXOSJ8o2W1NY6run6y68c10jSnW/NdFzz3/csCmpbuHZxBgQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVvAwUqAVc0VGBVfXtj7EnYTOrjPOe0vcXtMEneBKcQYEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFbwMFKgFSt545ag6v572MLQNhJCMx6f7rgm+vNtTdAJrhRnQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQ8jRasUFhsbXF17T4g7adiRu9Ic19w1/TPHNf+RuMBxzTlX53fTD04kOq6J2/KN45qzjitwNXAGBACwggACAFjhKIDy8/N12223KTY2VomJiRozZoxKSkoCxpw+fVq5ubnq1KmTYmJiNG7cOFVWVoa0aQBAy+cogIqKipSbm6utW7dq3bp1qqur0/Dhw1VTU+MfM3PmTH388cdasWKFioqKdOjQIY0dOzbkjQMAWjZHNyGsXbs24PWSJUuUmJioHTt2aMiQIaqqqtK//du/admyZbrzzjslSYsXL9ZNN92krVu36p/+6Z9C1zkAoEW7omtAVVVVkqSOHTtKknbs2KG6ujplZWX5x9x4443q0qWLtmzZ0uA6amtr5fV6AyYAQOsXdAD5fD7NmDFDAwcOVO/evSVJFRUVioqKUvv27QPGJiUlqaKiosH15Ofny+Px+Ke0NOe3pwIAWp6gAyg3N1e7d+/W8uXLr6iB2bNnq6qqyj+Vl5df0foAAC1DUB9EzcvL0+rVq7Vp0yZ17tzZPz85OVlnzpzR8ePHA86CKisrlZyc3OC63G633G53MG0AAFowR2dAxhjl5eVp5cqV2rhxo9LT0wOW9+vXT5GRkdqwYYN/XklJiQ4cOKABAwaEpmMAQKvg6AwoNzdXy5Yt06pVqxQbG+u/ruPxeNS2bVt5PB796le/0qxZs9SxY0fFxcVp+vTpGjBgAHfAAQACOAqghQsXSpKGDh0aMH/x4sWaNGmSJOn3v/+9wsLCNG7cONXW1io7O1tvvvlmSJoFALQeLmOMsd3E+bxerzwej4ZqtCJckbbbuSaE9b0pqLo9uTGOa5LT/q/jmiMlCY5rJt9Z6LhGkp7q9HVQdQhOn88nOa7p8ouvQt8IQuqsqVOhVqmqqkpxcXGNjuNZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAiqG9ERfPl6tfLcU3b3x8Jalv/nfFuUHWO9bk6m2nuak2d45pIV3hQ26qsr3VcM+dQjuOazq8F1x9aB86AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKHkbaytR0jXFc8373fw1ya1FB1jU9n3xB1c08NNhxzW8S1zuuyf4813FNbGG045rqbo5LJEnps7cEUVXtuCJMO4PYDloLzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoeRtrKRH+4zXHNLw4/HNS2jv64neMaXxDPL61z/nxVvf1//uC8SFLpbacd10z78VTHNek7dzmukTGOS+KdbwW4ajgDAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArXMYE8YTDJuT1euXxeDRUoxXhirTdDgDAobOmToVapaqqKsXFxTU6jjMgAIAVBBAAwApHAZSfn6/bbrtNsbGxSkxM1JgxY1RSUhIwZujQoXK5XAHTww8H930zAIDWy1EAFRUVKTc3V1u3btW6detUV1en4cOHq6amJmDclClTdPjwYf80b968kDYNAGj5HH0j6tq1awNeL1myRImJidqxY4eGDBninx8dHa3k5OTQdAgAaJWu6BpQVVWVJKljx44B85cuXar4+Hj17t1bs2fP1smTJxtdR21trbxeb8AEAGj9HJ0Bnc/n82nGjBkaOHCgevfu7Z9///33q2vXrkpNTdWuXbv01FNPqaSkRB9++GGD68nPz9eLL74YbBsAgBYq6M8BTZs2TZ988ok2b96szp07Nzpu48aNGjZsmPbt26eMjIwLltfW1qq2ttb/2uv1Ki0tjc8BAUALdbmfAwrqDCgvL0+rV6/Wpk2bLho+kpSZmSlJjQaQ2+2W2+0Opg0AQAvmKICMMZo+fbpWrlypwsJCpaenX7Jm586dkqSUlJSgGgQAtE6OAig3N1fLli3TqlWrFBsbq4qKCkmSx+NR27ZtVVpaqmXLlmnkyJHq1KmTdu3apZkzZ2rIkCHq06dPk/wAAICWydE1IJfL1eD8xYsXa9KkSSovL9eDDz6o3bt3q6amRmlpabr33nv17LPPXvR9wPPxLDgAaNma5BrQpbIqLS1NRUVFTlYJALhG8Sw4AIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVEbYb+CFjjCTprOokY7kZAIBjZ1Un6X/+P29Mswug6upqSdJmrbHcCQDgSlRXV8vj8TS63GUuFVFXmc/n06FDhxQbGyuXyxWwzOv1Ki0tTeXl5YqLi7PUoX3sh3PYD+ewH85hP5zTHPaDMUbV1dVKTU1VWFjjV3qa3RlQWFiYOnfufNExcXFx1/QB9j32wznsh3PYD+ewH86xvR8udubzPW5CAABYQQABAKxoUQHkdrs1Z84cud1u261YxX44h/1wDvvhHPbDOS1pPzS7mxAAANeGFnUGBABoPQggAIAVBBAAwAoCCABgBQEEALCixQTQggUL1K1bN7Vp00aZmZn64osvbLd01b3wwgtyuVwB04033mi7rSa3adMmjRo1SqmpqXK5XProo48Clhtj9PzzzyslJUVt27ZVVlaW9u7da6fZJnSp/TBp0qQLjo8RI0bYabaJ5Ofn67bbblNsbKwSExM1ZswYlZSUBIw5ffq0cnNz1alTJ8XExGjcuHGqrKy01HHTuJz9MHTo0AuOh4cffthSxw1rEQH0/vvva9asWZozZ46+/PJL9e3bV9nZ2Tpy5Ijt1q66Xr166fDhw/5p8+bNtltqcjU1Nerbt68WLFjQ4PJ58+bp9ddf16JFi7Rt2za1a9dO2dnZOn369FXutGldaj9I0ogRIwKOj/fee+8qdtj0ioqKlJubq61bt2rdunWqq6vT8OHDVVNT4x8zc+ZMffzxx1qxYoWKiop06NAhjR071mLXoXc5+0GSpkyZEnA8zJs3z1LHjTAtQP/+/U1ubq7/dX19vUlNTTX5+fkWu7r65syZY/r27Wu7DaskmZUrV/pf+3w+k5ycbF5++WX/vOPHjxu3223ee+89Cx1eHT/cD8YYM3HiRDN69Ggr/dhy5MgRI8kUFRUZY8793UdGRpoVK1b4x/zjH/8wksyWLVtstdnkfrgfjDHm9ttvN4899pi9pi5Dsz8DOnPmjHbs2KGsrCz/vLCwMGVlZWnLli0WO7Nj7969Sk1NVffu3fXAAw/owIEDtluyqqysTBUVFQHHh8fjUWZm5jV5fBQWFioxMVE9e/bUtGnTdOzYMdstNamqqipJUseOHSVJO3bsUF1dXcDxcOONN6pLly6t+nj44X743tKlSxUfH6/evXtr9uzZOnnypI32GtXsnob9Q99++63q6+uVlJQUMD8pKUl79uyx1JUdmZmZWrJkiXr27KnDhw/rxRdf1ODBg7V7927Fxsbabs+KiooKSWrw+Ph+2bVixIgRGjt2rNLT01VaWqpnnnlGOTk52rJli8LDw223F3I+n08zZszQwIED1bt3b0nnjoeoqCi1b98+YGxrPh4a2g+SdP/996tr165KTU3Vrl279NRTT6mkpEQffvihxW4DNfsAwv/Iycnx/7lPnz7KzMxU165d9cEHH+hXv/qVxc7QHIwfP97/55tvvll9+vRRRkaGCgsLNWzYMIudNY3c3Fzt3r37mrgOejGN7YeHHnrI/+ebb75ZKSkpGjZsmEpLS5WRkXG122xQs38LLj4+XuHh4RfcxVJZWank5GRLXTUP7du3V48ePbRv3z7brVjz/THA8XGh7t27Kz4+vlUeH3l5eVq9erUKCgoCvj8sOTlZZ86c0fHjxwPGt9bjobH90JDMzExJalbHQ7MPoKioKPXr108bNmzwz/P5fNqwYYMGDBhgsTP7Tpw4odLSUqWkpNhuxZr09HQlJycHHB9er1fbtm275o+PgwcP6tixY63q+DDGKC8vTytXrtTGjRuVnp4esLxfv36KjIwMOB5KSkp04MCBVnU8XGo/NGTnzp2S1LyOB9t3QVyO5cuXG7fbbZYsWWL+/ve/m4ceesi0b9/eVFRU2G7tqnr88cdNYWGhKSsrM5999pnJysoy8fHx5siRI7Zba1LV1dWmuLjYFBcXG0nm1VdfNcXFxeabb74xxhjzu9/9zrRv396sWrXK7Nq1y4wePdqkp6ebU6dOWe48tC62H6qrq80TTzxhtmzZYsrKysz69evNT37yE3PDDTeY06dP2249ZKZNm2Y8Ho8pLCw0hw8f9k8nT570j3n44YdNly5dzMaNG8327dvNgAEDzIABAyx2HXqX2g/79u0zL730ktm+fbspKyszq1atMt27dzdDhgyx3HmgFhFAxhjzxhtvmC5dupioqCjTv39/s3XrVtstXXX33XefSUlJMVFRUea6664z9913n9m3b5/ttppcQUGBkXTBNHHiRGPMuVuxn3vuOZOUlGTcbrcZNmyYKSkpsdt0E7jYfjh58qQZPny4SUhIMJGRkaZr165mypQpre6XtIZ+fklm8eLF/jGnTp0yjzzyiOnQoYOJjo429957rzl8+LC9ppvApfbDgQMHzJAhQ0zHjh2N2+02119/vfnNb35jqqqq7Db+A3wfEADAimZ/DQgA0DoRQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAV/w/hgVLrpVGHsAAAAABJRU5ErkJggg==", "text/plain": [ "
" ] @@ -1165,7 +1379,7 @@ }, { "cell_type": "code", - "execution_count": 38, + "execution_count": 40, "id": "f1285e8b-1b96-437b-973a-eb868e33afb7", "metadata": {}, "outputs": [], @@ -1179,13 +1393,22 @@ }, { "cell_type": "code", - "execution_count": 39, + "execution_count": 41, "id": "6bea332e-f6de-494f-a0db-795d9fe3e134", "metadata": {}, "outputs": [], "source": [ "def predict_batch_fn():\n", " import tensorflow as tf\n", + " # Enable GPU memory growth\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + " \n", " model = tf.keras.models.load_model(model_dir)\n", " def predict(inputs: np.ndarray) -> np.ndarray:\n", " return model.predict(inputs)\n", @@ -1195,7 +1418,7 @@ }, { "cell_type": "code", - "execution_count": 40, + "execution_count": 42, "id": "731d234c-549f-4df3-8a2b-312e63195396", "metadata": {}, "outputs": [], @@ -1208,7 +1431,7 @@ }, { "cell_type": "code", - "execution_count": 41, + "execution_count": 43, "id": "a40fe207-6246-4b0e-abde-823979878d97", "metadata": {}, "outputs": [ @@ -1218,7 +1441,7 @@ "784" ] }, - "execution_count": 41, + "execution_count": 43, "metadata": {}, "output_type": "execute_result" } @@ -1230,7 +1453,7 @@ }, { "cell_type": "code", - "execution_count": 42, + "execution_count": 44, "id": "10904f12-03e7-4518-8f12-2aa11989ddf5", "metadata": {}, "outputs": [ @@ -1238,15 +1461,22 @@ "name": "stderr", "output_type": "stream", "text": [ - " \r" + "[Stage 10:=======> (1 + 7) / 8]\r" ] }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 971 ms, sys: 83 ms, total: 1.05 s\n", - "Wall time: 8.63 s\n" + "CPU times: user 45.6 ms, sys: 26 ms, total: 71.6 ms\n", + "Wall time: 5.51 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" ] } ], @@ -1257,23 +1487,16 @@ }, { "cell_type": "code", - "execution_count": 43, + "execution_count": 45, "id": "671128df-f0f4-4f54-b35c-d63a78c7f89a", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 823 ms, sys: 136 ms, total: 959 ms\n", - "Wall time: 3.45 s\n" + "CPU times: user 46.5 ms, sys: 34 ms, total: 80.5 ms\n", + "Wall time: 884 ms\n" ] } ], @@ -1284,7 +1507,7 @@ }, { "cell_type": "code", - "execution_count": 44, + "execution_count": 46, "id": "ce35deaf-7d49-4f34-9bf9-b4e6fc5761f4", "metadata": {}, "outputs": [], @@ -1303,10 +1526,17 @@ }, { "cell_type": "code", - "execution_count": 45, + "execution_count": 47, "id": "f9119632-b284-45d7-a262-c262e034c15c", "metadata": {}, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, { "data": { "text/html": [ @@ -1374,7 +1604,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [-9.66701889038086, -3.6094393730163574, -3.31...\n", + " [-5.88436, -3.1058552, 0.108737305, 12.67319, ...\n", " \n", " \n", " 1\n", @@ -1398,7 +1628,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [-16.348304748535156, 3.1133787631988525, 6.73...\n", + " [-3.2732859, -8.362555, 1.893612, -3.888143, 0...\n", " \n", " \n", " 2\n", @@ -1422,7 +1652,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [-21.153539657592773, -6.192655086517334, -6.6...\n", + " [-3.3856308, 0.6785604, 1.3146865, 0.9275978, ...\n", " \n", " \n", " 3\n", @@ -1446,7 +1676,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [-13.79001522064209, -16.20423126220703, -8.87...\n", + " [-2.775469, -7.3659234, 11.768431, 1.3434289, ...\n", " \n", " \n", " 4\n", @@ -1470,7 +1700,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [-18.723485946655273, -7.806203365325928, -8.6...\n", + " [-4.942663, 4.0774136, -0.45292768, -0.9312788...\n", " \n", " \n", " 5\n", @@ -1494,7 +1724,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [10.495272636413574, -10.992117881774902, -2.4...\n", + " [-5.226616, -3.1389174, 2.6100307, 3.695045, -...\n", " \n", " \n", " 6\n", @@ -1518,7 +1748,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [-9.437854766845703, -3.7454347610473633, -3.5...\n", + " [-4.3006196, 5.116993, 0.5850617, -0.7624871, ...\n", " \n", " \n", " 7\n", @@ -1542,7 +1772,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [-12.314886093139648, -2.106672763824463, -0.7...\n", + " [-2.398596, -1.4814726, -4.8840575, -0.2391601...\n", " \n", " \n", " 8\n", @@ -1566,7 +1796,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [-22.142608642578125, -2.8917746543884277, -4....\n", + " [0.82160157, -2.8640628, -1.6951559, -4.489291...\n", " \n", " \n", " 9\n", @@ -1590,7 +1820,7 @@ " 0.0\n", " 0.0\n", " 0.0\n", - " [-12.939343452453613, 0.40699127316474915, 15....\n", + " [-1.2338604, -2.151981, -4.1717424, 1.6106843,...\n", " \n", " \n", "\n", @@ -1598,8 +1828,8 @@ "" ], "text/plain": [ - " 0 1 2 3 4 5 6 7 8 9 ... 775 776 777 778 \n", - "0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \\\n", + " 0 1 2 3 4 5 6 7 8 9 ... 775 776 777 778 \\\n", + "0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", "1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", "2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", "3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", @@ -1611,21 +1841,21 @@ "9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", "\n", " 779 780 781 782 783 preds \n", - "0 0.0 0.0 0.0 0.0 0.0 [-9.66701889038086, -3.6094393730163574, -3.31... \n", - "1 0.0 0.0 0.0 0.0 0.0 [-16.348304748535156, 3.1133787631988525, 6.73... \n", - "2 0.0 0.0 0.0 0.0 0.0 [-21.153539657592773, -6.192655086517334, -6.6... \n", - "3 0.0 0.0 0.0 0.0 0.0 [-13.79001522064209, -16.20423126220703, -8.87... \n", - "4 0.0 0.0 0.0 0.0 0.0 [-18.723485946655273, -7.806203365325928, -8.6... \n", - "5 0.0 0.0 0.0 0.0 0.0 [10.495272636413574, -10.992117881774902, -2.4... \n", - "6 0.0 0.0 0.0 0.0 0.0 [-9.437854766845703, -3.7454347610473633, -3.5... \n", - "7 0.0 0.0 0.0 0.0 0.0 [-12.314886093139648, -2.106672763824463, -0.7... \n", - "8 0.0 0.0 0.0 0.0 0.0 [-22.142608642578125, -2.8917746543884277, -4.... \n", - "9 0.0 0.0 0.0 0.0 0.0 [-12.939343452453613, 0.40699127316474915, 15.... \n", + "0 0.0 0.0 0.0 0.0 0.0 [-5.88436, -3.1058552, 0.108737305, 12.67319, ... \n", + "1 0.0 0.0 0.0 0.0 0.0 [-3.2732859, -8.362555, 1.893612, -3.888143, 0... \n", + "2 0.0 0.0 0.0 0.0 0.0 [-3.3856308, 0.6785604, 1.3146865, 0.9275978, ... \n", + "3 0.0 0.0 0.0 0.0 0.0 [-2.775469, -7.3659234, 11.768431, 1.3434289, ... \n", + "4 0.0 0.0 0.0 0.0 0.0 [-4.942663, 4.0774136, -0.45292768, -0.9312788... \n", + "5 0.0 0.0 0.0 0.0 0.0 [-5.226616, -3.1389174, 2.6100307, 3.695045, -... \n", + "6 0.0 0.0 0.0 0.0 0.0 [-4.3006196, 5.116993, 0.5850617, -0.7624871, ... \n", + "7 0.0 0.0 0.0 0.0 0.0 [-2.398596, -1.4814726, -4.8840575, -0.2391601... \n", + "8 0.0 0.0 0.0 0.0 0.0 [0.82160157, -2.8640628, -1.6951559, -4.489291... \n", + "9 0.0 0.0 0.0 0.0 0.0 [-1.2338604, -2.151981, -4.1717424, 1.6106843,... \n", "\n", "[10 rows x 785 columns]" ] }, - "execution_count": 45, + "execution_count": 47, "metadata": {}, "output_type": "execute_result" } @@ -1637,7 +1867,7 @@ }, { "cell_type": "code", - "execution_count": 46, + "execution_count": 48, "id": "7c067c62-03a6-461e-a1ff-4653276fbea1", "metadata": {}, "outputs": [], @@ -1648,26 +1878,19 @@ }, { "cell_type": "code", - "execution_count": 47, + "execution_count": 49, "id": "a7084ad0-c021-4296-bad0-7a238971f53b", "metadata": {}, "outputs": [ { "data": { "text/plain": [ - "[-9.66701889038086,\n", - " -3.6094393730163574,\n", - " -3.3102951049804688,\n", - " -0.32629871368408203,\n", - " -8.564104080200195,\n", - " -3.614396095275879,\n", - " -5.390956878662109,\n", - " -3.0429909229278564,\n", - " 9.717466354370117,\n", - " -5.417511463165283]" + "array([ -5.88436 , -3.1058552, 0.1087373, 12.67319 , -5.1437874,\n", + " 4.085992 , -10.203137 , -1.4333997, -3.3865087, -3.8473575],\n", + " dtype=float32)" ] }, - "execution_count": 47, + "execution_count": 49, "metadata": {}, "output_type": "execute_result" } @@ -1679,7 +1902,7 @@ }, { "cell_type": "code", - "execution_count": 48, + "execution_count": 50, "id": "8167c832-93ef-4f50-873b-07b67c19ef53", "metadata": {}, "outputs": [], @@ -1691,13 +1914,13 @@ }, { "cell_type": "code", - "execution_count": 49, + "execution_count": 51, "id": "297811e1-aecb-4afd-9a6a-30c49e8881cc", "metadata": {}, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAk1klEQVR4nO3dfXRU9b3v8c/kaXhKBkKeJWBAHixIrBQihwdRUkJQFk/HIz7cBegBxIAgtXpiVUS5Ny2eemhpBM+qEm1BkNMiS0tRnhJEgVMQDherKeRECRcSkCszEEiA5Hf/4DJ1IAH3OMkvCe/XWnstZu/fd/aXzYYPe/aeX1zGGCMAABpZmO0GAADXJwIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIcODGG2/U5MmT/a8LCwvlcrlUWFgYsn24XC698MILIXs/oKkigNBsFBQUyOVy+ZdWrVqpR48emjlzpioqKmy358i6deuaVci88847uv3229W+fXt17NhRd9xxh/70pz/ZbgvNXITtBgCnXnzxRaWlpamqqkrbtm3TkiVLtG7dOu3fv19t2rRp1F6GDh2qs2fPKioqylHdunXrlJ+fX2cInT17VhERTeev5uLFi/X444/r7rvv1s9//nNVVVWpoKBA99xzj/7whz9o/PjxtltEM9V0znLgO8rOztaPfvQjSdI///M/q2PHjnrllVe0du1a3X///XXWVFZWqm3btiHvJSwsTK1atQrpe4b6/b6vxYsXq3///nrvvffkcrkkSQ8//LBuuOEGvfnmmwQQgsZHcGj27rrrLklSaWmpJGny5Mlq166dSkpKNGrUKEVHR+vBBx+UJNXW1mrRokXq3bu3WrVqpcTERE2fPl3ffPNNwHsaY7RgwQJ16tRJbdq00Z133qnPPvvsin3Xdw9o586dGjVqlDp06KC2bduqb9+++tWvfuXvLz8/X5ICPlK8pK57QHv27FF2drZiYmLUrl07DR8+XDt27AgYc+kjyo8//lhz585VfHy82rZtq3Hjxun48eMBY71er7744gt5vd5rHl+fz6eEhISAHi/10bp162vWA/XhCgjNXklJiSSpY8eO/nUXLlxQVlaWBg8erH/913/1fzQ3ffp0FRQUaMqUKXr88cdVWlqq3/zmN9qzZ48+/vhjRUZGSpKef/55LViwQKNGjdKoUaP06aefasSIETp37tw1+9mwYYPuueceJScna/bs2UpKStLnn3+u999/X7Nnz9b06dN15MgRbdiwQb/73e+u+X6fffaZhgwZopiYGD311FOKjIzUa6+9pmHDhqmoqEgZGRkB42fNmqUOHTpo3rx5+vLLL7Vo0SLNnDlTq1at8o9Zs2aNpkyZomXLlgU8VFGXYcOG6T/+4z+0ePFijR49WlVVVVq8eLG8Xq9mz559zf6BehmgmVi2bJmRZDZu3GiOHz9uysrKzMqVK03Hjh1N69atzeHDh40xxkyaNMlIMv/yL/8SUP/RRx8ZSWb58uUB69evXx+w/tixYyYqKsrcfffdpra21j/umWeeMZLMpEmT/Ou2bNliJJktW7YYY4y5cOGCSUtLM126dDHffPNNwH6+/V45OTmmvr9+ksy8efP8r8eOHWuioqJMSUmJf92RI0dMdHS0GTp06BXHJzMzM2BfTzzxhAkPDzcnT568YuyyZcvq7OHbKioqzPDhw40k/xIXF2c++eSTa9YCV8NHcGh2MjMzFR8fr9TUVE2cOFHt2rXTmjVrdMMNNwSMmzFjRsDr1atXy+Px6Mc//rG+/vpr/9KvXz+1a9dOW7ZskSRt3LhR586d06xZswI+dpozZ841e9uzZ49KS0s1Z84ctW/fPmDbt9/ru6qpqdGHH36osWPHqmvXrv71ycnJeuCBB7Rt2zb5fL6AmmnTpgXsa8iQIaqpqdFXX33lXzd58mQZY6559SNJbdq0Uc+ePTVp0iStXr1ab7zxhpKTkzV+/HgdPHjQ8e8JuISP4NDs5Ofnq0ePHoqIiFBiYqJ69uypsLDA/0tFRESoU6dOAesOHDggr9erhISEOt/32LFjkuT/h7p79+4B2+Pj49WhQ4er9nbp48A+ffp899/QVRw/flxnzpxRz549r9h28803q7a2VmVlZerdu7d/fefOnQPGXer58vtc39W9996riIgIvffee/51Y8aMUffu3fWzn/0s4KM9wAkCCM3OgAED/E/B1cftdl8RSrW1tUpISNDy5cvrrImPjw9ZjzaFh4fXud4Y4/i9/vu//1vr16/Xv//7vwesj42N1eDBg/Xxxx8H1SMgEUC4jnTr1k0bN27UoEGDrvr0VpcuXSRdvGL69sdex48fv+ZVRLdu3SRJ+/fvV2ZmZr3jvuvHcfHx8WrTpo2Ki4uv2PbFF18oLCxMqamp3+m9gnHpC741NTVXbDt//rwuXLjQYPtGy8c9IFw3/umf/kk1NTV66aWXrth24cIFnTx5UtLFe0yRkZFavHhxwFXDokWLrrmP2267TWlpaVq0aJH//S759ntd+k7S5WMuFx4erhEjRmjt2rX68ssv/esrKiq0YsUKDR48WDExMdfs63Lf9THsm266SWFhYVq1alVA/4cPH9ZHH32kH/7wh473DVzCFRCuG3fccYemT5+uvLw87d27VyNGjFBkZKQOHDig1atX61e/+pX+8R//UfHx8XryySeVl5ene+65R6NGjdKePXv05z//WXFxcVfdR1hYmJYsWaLRo0fr1ltv1ZQpU5ScnKwvvvhCn332mT744ANJUr9+/SRJjz/+uLKyshQeHq6JEyfW+Z4LFizQhg0bNHjwYD322GOKiIjQa6+9purqai1cuDCoY/FdH8OOj4/Xww8/rN/+9rcaPny4xo8fr1OnTunVV1/V2bNnlZubG9T+AYkAwnVm6dKl6tevn1577TU988wzioiI0I033qiHHnpIgwYN8o9bsGCBWrVqpaVLl2rLli3KyMjQhx9+qLvvvvua+8jKytKWLVs0f/58/fKXv1Rtba26deumqVOn+seMHz9es2bN0sqVK/X73/9exph6A6h379766KOPlJubq7y8PNXW1iojI0O///3vr/gOUENYsmSJ0tPT9frrr/sDp3///nrrrbc0dOjQBt8/Wi6XCebOJAAA3xP3gAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsKLJfQ+otrZWR44cUXR0dFCzBwMA7DLG6NSpU0pJSbliTsZva3IBdOTIkQad2woA0DjKysqumJX+25pcAEVHR0uSBmuUIhRpuRsAgFMXdF7btM7/73l9GiyA8vPz9fLLL6u8vFzp6elavHixBgwYcM26Sx+7RShSES4CCACanf8/v861bqM0yEMIq1at0ty5czVv3jx9+umnSk9PV1ZWlv8HfgEA0CAB9Morr2jq1KmaMmWKfvCDH2jp0qVq06aN3njjjYbYHQCgGQp5AJ07d067d+8O+GFcYWFhyszM1Pbt268YX11dLZ/PF7AAAFq+kAfQ119/rZqaGiUmJgasT0xMVHl5+RXj8/Ly5PF4/AtPwAHA9cH6F1Fzc3Pl9Xr9S1lZme2WAACNIORPwcXFxSk8PNz/s+QvqaioUFJS0hXj3W633G53qNsAADRxIb8CioqKUr9+/bRp0yb/utraWm3atEkDBw4M9e4AAM1Ug3wPaO7cuZo0aZJ+9KMfacCAAVq0aJEqKys1ZcqUhtgdAKAZapAAuu+++3T8+HE9//zzKi8v16233qr169df8WACAOD65TLGGNtNfJvP55PH49EwjWEmBABohi6Y8yrUWnm9XsXExNQ7zvpTcACA6xMBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArIiw3QBwLeExMY5r/vZc76D29XDWZsc1z8QVO66pMbWOa768cMZxzagdjzmukaQui1yOa1zb/yuofeH6xRUQAMAKAggAYEXIA+iFF16Qy+UKWHr16hXq3QAAmrkGuQfUu3dvbdy48e87ieBWEwAgUIMkQ0REhJKSkhrirQEALUSD3AM6cOCAUlJS1LVrVz344IM6dOhQvWOrq6vl8/kCFgBAyxfyAMrIyFBBQYHWr1+vJUuWqLS0VEOGDNGpU6fqHJ+XlyePx+NfUlNTQ90SAKAJCnkAZWdn695771Xfvn2VlZWldevW6eTJk3rnnXfqHJ+bmyuv1+tfysrKQt0SAKAJavCnA9q3b68ePXro4MGDdW53u91yu90N3QYAoIlp8O8BnT59WiUlJUpOTm7oXQEAmpGQB9CTTz6poqIiffnll/rkk080btw4hYeH6/777w/1rgAAzVjIP4I7fPiw7r//fp04cULx8fEaPHiwduzYofj4+FDvCgDQjLmMMcZ2E9/m8/nk8Xg0TGMU4Yq03Q5CzAy61XHNi7973XFNf7fzyTTxd3+o7OC4piDjNsc1Nd9847gGTd8Fc16FWiuv16uYq0wmzFxwAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFg/9AOuDbXC9+7bgmmIlFV5/u6LhGkl7+248d10Suig1qX051ml73D3W8mlVdPwxqX2dqg/ghkTU1Qe0L1y+ugAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFs2GjUZ1eeoPzokXOS07WtHFeJCnxyVrHNTXFO4Lal1Nn/tTBcc2QrMeC2leU1/nM1m7fX4LaV2NwuYOY3VtSWLu2jmtqTvzfoPZ1PeIKCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsYDJSNCrPB587rll5Ot5xzVRPmeMaSTq0sqPjmt0/bJz/x9V8843jmuiVjTNRamNy9b/FcY33xTNB7Su+TaXjmpo7gtrVdYkrIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwgslI0ahqfD7HNb97aJTjmttWv+a4RpLmxm13XNP/tTmOa3pM/4vjmqYurFUrxzUl837ouGbnQ790XFMj47hGkob89qeOazqrPKh9XY+4AgIAWEEAAQCscBxAW7du1ejRo5WSkiKXy6V33303YLsxRs8//7ySk5PVunVrZWZm6sCBA6HqFwDQQjgOoMrKSqWnpys/P7/O7QsXLtSvf/1rLV26VDt37lTbtm2VlZWlqqqq790sAKDlcPwQQnZ2trKzs+vcZozRokWL9Oyzz2rMmDGSpLfeekuJiYl69913NXHixO/XLQCgxQjpPaDS0lKVl5crMzPTv87j8SgjI0Pbt9f9dFF1dbV8Pl/AAgBo+UIaQOXlFx8/TExMDFifmJjo33a5vLw8eTwe/5KamhrKlgAATZT1p+Byc3Pl9Xr9S1lZme2WAACNIKQBlJSUJEmqqKgIWF9RUeHfdjm3262YmJiABQDQ8oU0gNLS0pSUlKRNmzb51/l8Pu3cuVMDBw4M5a4AAM2c46fgTp8+rYMHD/pfl5aWau/evYqNjVXnzp01Z84cLViwQN27d1daWpqee+45paSkaOzYsaHsGwDQzDkOoF27dunOO+/0v547d64kadKkSSooKNBTTz2lyspKTZs2TSdPntTgwYO1fv16tQpinigAQMvlMsYEN0tfA/H5fPJ4PBqmMYpwRdpuB83V7X2DKlu+eonjmtog9pOxbo7jmmAmMA1mglBJKn/kNsc1907fdO1Bl3m64+eOa/JO/MBxTeHsf3BcI0nhWz4Nqu56d8GcV6HWyuv1XvW+vvWn4AAA1ycCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsYDZs4FvOjM9wXFO42PkM2idqzzquufPVnzquaTv4uOMaSdp+66qg6pzq88kkxzU3/o8Djmtqq6oc1yB4zIYNAGjSCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFhO0GgKak3br/clzTfdgMxzUHJjifwHTfzN84rglWMJOl/sM7P3Fc0/1nex3XMLFoy8EVEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYwWSkwLcYYxzXtPsqvAE6CY1gJhWVpJH/60nHNd2WbHdcU+u4Ai0JV0AAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAWTkaJFcrndQdUd/J8/dFxTfP9vgtpXY/jtN7cFVRcfxMSigFNcAQEArCCAAABWOA6grVu3avTo0UpJSZHL5dK7774bsH3y5MlyuVwBy8iRI0PVLwCghXAcQJWVlUpPT1d+fn69Y0aOHKmjR4/6l7fffvt7NQkAaHkcP4SQnZ2t7Ozsq45xu91KSkoKuikAQMvXIPeACgsLlZCQoJ49e2rGjBk6ceJEvWOrq6vl8/kCFgBAyxfyABo5cqTeeustbdq0Sb/4xS9UVFSk7Oxs1dTU1Dk+Ly9PHo/Hv6Smpoa6JQBAExTy7wFNnDjR/+tbbrlFffv2Vbdu3VRYWKjhw4dfMT43N1dz5871v/b5fIQQAFwHGvwx7K5duyouLk4HDx6sc7vb7VZMTEzAAgBo+Ro8gA4fPqwTJ04oOTm5oXcFAGhGHH8Ed/r06YCrmdLSUu3du1exsbGKjY3V/PnzNWHCBCUlJamkpERPPfWUbrrpJmVlZYW0cQBA8+Y4gHbt2qU777zT//rS/ZtJkyZpyZIl2rdvn958802dPHlSKSkpGjFihF566SW5g5ybCwDQMjkOoGHDhskYU+/2Dz744Hs1BITC8cnBTcJZfH/9X7AOpdWnOzquubdd/V9nqM+0Dp86rpGkbb0fdlxT81lxUPvC9Yu54AAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFyH8kNxBq4R06OK6Z9+SbDdBJ3fJO/MBxzdZHMxzX3Lt6meOaDmGtHddI0tlO0Y5roj4Lale4jnEFBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWMBkpmrzy+3o5rrm7zaag9rXprNtxzSc/7uy4JqzTBcc1jelsnPN/GqIaoA+0bFwBAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVTEaKRhUeE+O45qUnlzVAJ3XLKx3luCaq4ivHNa5O8Y5rgnHowpmg6mL/ctxxTU1Qe8L1jCsgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCyUjRqI4+1MdxzcjWhY5r/na+ynGNJLWe7XZcY6KjHdec+8UpxzXBeNvbL6i6mr+VhLgT4EpcAQEArCCAAABWOAqgvLw89e/fX9HR0UpISNDYsWNVXFwcMKaqqko5OTnq2LGj2rVrpwkTJqiioiKkTQMAmj9HAVRUVKScnBzt2LFDGzZs0Pnz5zVixAhVVlb6xzzxxBN67733tHr1ahUVFenIkSMaP358yBsHADRvjh5CWL9+fcDrgoICJSQkaPfu3Ro6dKi8Xq9ef/11rVixQnfddZckadmyZbr55pu1Y8cO3X777aHrHADQrH2ve0Ber1eSFBsbK0navXu3zp8/r8zMTP+YXr16qXPnztq+fXud71FdXS2fzxewAABavqADqLa2VnPmzNGgQYPUp8/FR2vLy8sVFRWl9u3bB4xNTExUeXl5ne+Tl5cnj8fjX1JTU4NtCQDQjAQdQDk5Odq/f79Wrlz5vRrIzc2V1+v1L2VlZd/r/QAAzUNQX0SdOXOm3n//fW3dulWdOnXyr09KStK5c+d08uTJgKugiooKJSUl1flebrdbbrfzL/8BAJo3R1dAxhjNnDlTa9as0ebNm5WWlhawvV+/foqMjNSmTZv864qLi3Xo0CENHDgwNB0DAFoER1dAOTk5WrFihdauXavo6Gj/fR2Px6PWrVvL4/HokUce0dy5cxUbG6uYmBjNmjVLAwcO5Ak4AEAARwG0ZMkSSdKwYcMC1i9btkyTJ0+WJP3bv/2bwsLCNGHCBFVXVysrK0uvvvpqSJoFALQcjgLIGHPNMa1atVJ+fr7y8/ODbgrNQ1jbto5rFsx9owE6udJKb/+g6mr++jfHNWXP/oPjmv99828c1xy6cMZxzZ9eGua4RpLaaWdQdYATzAUHALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAK4L6iaiAJJkLFxzX/LXqBsc1I1sfcFzzwf+52XGNJNVOindcs+vRRUHsyflfvbv+NNdxTY/VzGqNposrIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwgslIETRTXe24pmB5luOauTOdT0b6UfoqxzWSVJte67imouac45o7i6Y7rukxc7fjGqAp4woIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKxgMlI0qi5LP3dcM+++dMc18+P/y3GNJN3zxTjHNTUvJTiuuanwU8c1QEvDFRAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWMFkpGhUNd9847jmL7eGO64Zpdsc11x02HFFeBA1ALgCAgBYQgABAKxwFEB5eXnq37+/oqOjlZCQoLFjx6q4uDhgzLBhw+RyuQKWRx99NKRNAwCaP0cBVFRUpJycHO3YsUMbNmzQ+fPnNWLECFVWVgaMmzp1qo4ePepfFi5cGNKmAQDNn6OHENavXx/wuqCgQAkJCdq9e7eGDh3qX9+mTRslJSWFpkMAQIv0ve4Beb1eSVJsbGzA+uXLlysuLk59+vRRbm6uzpw5U+97VFdXy+fzBSwAgJYv6Mewa2trNWfOHA0aNEh9+vTxr3/ggQfUpUsXpaSkaN++fXr66adVXFysP/7xj3W+T15enubPnx9sGwCAZspljDHBFM6YMUN//vOftW3bNnXq1KnecZs3b9bw4cN18OBBdevW7Yrt1dXVqq6u9r/2+XxKTU3VMI1RhCsymNYAABZdMOdVqLXyer2KiYmpd1xQV0AzZ87U+++/r61bt141fCQpIyNDkuoNILfbLbfbHUwbAIBmzFEAGWM0a9YsrVmzRoWFhUpLS7tmzd69eyVJycnJQTUIAGiZHAVQTk6OVqxYobVr1yo6Olrl5eWSJI/Ho9atW6ukpEQrVqzQqFGj1LFjR+3bt09PPPGEhg4dqr59+zbIbwAA0Dw5ugfkcrnqXL9s2TJNnjxZZWVleuihh7R//35VVlYqNTVV48aN07PPPnvVzwG/zefzyePxcA8IAJqpBrkHdK2sSk1NVVFRkZO3BABcp5gLDgBgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgRYTtBi5njJEkXdB5yVhuBgDg2AWdl/T3f8/r0+QC6NSpU5KkbVpnuRMAwPdx6tQpeTyeere7zLUiqpHV1tbqyJEjio6OlsvlCtjm8/mUmpqqsrIyxcTEWOrQPo7DRRyHizgOF3EcLmoKx8EYo1OnTiklJUVhYfXf6WlyV0BhYWHq1KnTVcfExMRc1yfYJRyHizgOF3EcLuI4XGT7OFztyucSHkIAAFhBAAEArGhWAeR2uzVv3jy53W7brVjFcbiI43ARx+EijsNFzek4NLmHEAAA14dmdQUEAGg5CCAAgBUEEADACgIIAGAFAQQAsKLZBFB+fr5uvPFGtWrVShkZGfrP//xP2y01uhdeeEEulytg6dWrl+22GtzWrVs1evRopaSkyOVy6d133w3YbozR888/r+TkZLVu3VqZmZk6cOCAnWYb0LWOw+TJk684P0aOHGmn2QaSl5en/v37Kzo6WgkJCRo7dqyKi4sDxlRVVSknJ0cdO3ZUu3btNGHCBFVUVFjquGF8l+MwbNiwK86HRx991FLHdWsWAbRq1SrNnTtX8+bN06effqr09HRlZWXp2LFjtltrdL1799bRo0f9y7Zt22y31OAqKyuVnp6u/Pz8OrcvXLhQv/71r7V06VLt3LlTbdu2VVZWlqqqqhq504Z1reMgSSNHjgw4P95+++1G7LDhFRUVKScnRzt27NCGDRt0/vx5jRgxQpWVlf4xTzzxhN577z2tXr1aRUVFOnLkiMaPH2+x69D7LsdBkqZOnRpwPixcuNBSx/UwzcCAAQNMTk6O/3VNTY1JSUkxeXl5FrtqfPPmzTPp6em227BKklmzZo3/dW1trUlKSjIvv/yyf93JkyeN2+02b7/9toUOG8flx8EYYyZNmmTGjBljpR9bjh07ZiSZoqIiY8zFP/vIyEizevVq/5jPP//cSDLbt2+31WaDu/w4GGPMHXfcYWbPnm2vqe+gyV8BnTt3Trt371ZmZqZ/XVhYmDIzM7V9+3aLndlx4MABpaSkqGvXrnrwwQd16NAh2y1ZVVpaqvLy8oDzw+PxKCMj47o8PwoLC5WQkKCePXtqxowZOnHihO2WGpTX65UkxcbGSpJ2796t8+fPB5wPvXr1UufOnVv0+XD5cbhk+fLliouLU58+fZSbm6szZ87YaK9eTW427Mt9/fXXqqmpUWJiYsD6xMREffHFF5a6siMjI0MFBQXq2bOnjh49qvnz52vIkCHav3+/oqOjbbdnRXl5uSTVeX5c2na9GDlypMaPH6+0tDSVlJTomWeeUXZ2trZv367w8HDb7YVcbW2t5syZo0GDBqlPnz6SLp4PUVFRat++fcDYlnw+1HUcJOmBBx5Qly5dlJKSon379unpp59WcXGx/vjHP1rsNlCTDyD8XXZ2tv/Xffv2VUZGhrp06aJ33nlHjzzyiMXO0BRMnDjR/+tbbrlFffv2Vbdu3VRYWKjhw4db7Kxh5OTkaP/+/dfFfdCrqe84TJs2zf/rW265RcnJyRo+fLhKSkrUrVu3xm6zTk3+I7i4uDiFh4df8RRLRUWFkpKSLHXVNLRv3149evTQwYMHbbdizaVzgPPjSl27dlVcXFyLPD9mzpyp999/X1u2bAn4+WFJSUk6d+6cTp48GTC+pZ4P9R2HumRkZEhSkzofmnwARUVFqV+/ftq0aZN/XW1trTZt2qSBAwda7My+06dPq6SkRMnJybZbsSYtLU1JSUkB54fP59POnTuv+/Pj8OHDOnHiRIs6P4wxmjlzptasWaPNmzcrLS0tYHu/fv0UGRkZcD4UFxfr0KFDLep8uNZxqMvevXslqWmdD7afgvguVq5cadxutykoKDB//etfzbRp00z79u1NeXm57dYa1U9+8hNTWFhoSktLzccff2wyMzNNXFycOXbsmO3WGtSpU6fMnj17zJ49e4wk88orr5g9e/aYr776yhhjzM9//nPTvn17s3btWrNv3z4zZswYk5aWZs6ePWu589C62nE4deqUefLJJ8327dtNaWmp2bhxo7nttttM9+7dTVVVle3WQ2bGjBnG4/GYwsJCc/ToUf9y5swZ/5hHH33UdO7c2WzevNns2rXLDBw40AwcONBi16F3reNw8OBB8+KLL5pdu3aZ0tJSs3btWtO1a1czdOhQy50HahYBZIwxixcvNp07dzZRUVFmwIABZseOHbZbanT33XefSU5ONlFRUeaGG24w9913nzl48KDtthrcli1bjKQrlkmTJhljLj6K/dxzz5nExETjdrvN8OHDTXFxsd2mG8DVjsOZM2fMiBEjTHx8vImMjDRdunQxU6dObXH/Savr9y/JLFu2zD/m7Nmz5rHHHjMdOnQwbdq0MePGjTNHjx6113QDuNZxOHTokBk6dKiJjY01brfb3HTTTeanP/2p8Xq9dhu/DD8PCABgRZO/BwQAaJkIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCK/wcgFXDPfhuVLAAAAABJRU5ErkJggg==", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAkfElEQVR4nO3dfXQUdZ7v8U/nqSEkaR7yLAFCFHRAcAYly/AgSiQEZUCYGUG9F7gziJiggI6KR0Udzsksrg7qIHjcHVhHEGWOyMoiDg9JGBRwwTCIM2QhJ0g4kIBcSYcAIaR/9w+uvbQkQDUdfkl4v86pc+iq37fqm6Lgk+qqrnYZY4wAALjKwmw3AAC4NhFAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAgAPdunXTpEmT/K8LCwvlcrlUWFgYsm24XC698MILIVsf0FwRQGgxlixZIpfL5Z/atGmjHj16KC8vT5WVlbbbc2TNmjUtJmTefvtt3X777UpKSpLb7VZ6eromT56s/fv3224NLVyE7QYAp1566SWlp6fr9OnT2rx5sxYuXKg1a9Zo9+7dio6Ovqq9DBkyRKdOnVJUVJSjujVr1mjBggUNhtCpU6cUEdF8/mkWFxcrPT1dP/vZz9ShQweVlZXp7bff1urVq/W3v/1NqamptltEC9V8jnLgMuXk5OjWW2+VJP36179Wp06d9Oqrr2rVqlWaMGFCgzU1NTVq165dyHsJCwtTmzZtQrrOUK/vSr355psXzBszZoxuvfVWvfPOO3r66actdIXWgLfg0OLdeeedkqSysjJJ0qRJkxQTE6PS0lKNHDlSsbGxeuCBByRJPp9P8+fPV69evdSmTRslJSVp6tSp+u677wLWaYzR3Llz1blzZ0VHR+uOO+7Q119/fcG2G7sGtG3bNo0cOVIdOnRQu3bt1KdPH7322mv+/hYsWCBJAW8pfq+ha0DFxcXKyclRXFycYmJiNGzYMG3dujVgzPdvUX722WeaNWuWEhIS1K5dO9177706evRowNiqqirt2bNHVVVVl7OLL9CtWzdJ0vHjx4OqByTOgNAKlJaWSpI6derkn3f27FllZ2dr0KBB+pd/+Rf/W3NTp07VkiVLNHnyZD366KMqKyvTH/7wBxUXF+uzzz5TZGSkJOn555/X3LlzNXLkSI0cOVJffvmlhg8frjNnzlyyn3Xr1umee+5RSkqKHnvsMSUnJ+sf//iHVq9erccee0xTp07VoUOHtG7dOv3pT3+65Pq+/vprDR48WHFxcXryyScVGRmpt956S0OHDlVRUZEyMzMDxk+fPl0dOnTQnDlztH//fs2fP195eXl6//33/WNWrlypyZMna/HixQE3VVzMsWPHVF9frwMHDuill16SJA0bNuyyaoEGGaCFWLx4sZFk1q9fb44ePWrKy8vN8uXLTadOnUzbtm3NwYMHjTHGTJw40UgyTz/9dED9X//6VyPJLF26NGD+2rVrA+YfOXLEREVFmbvvvtv4fD7/uGeeecZIMhMnTvTPKygoMJJMQUGBMcaYs2fPmvT0dNO1a1fz3XffBWzn/HXl5uaaxv75STJz5szxvx4zZoyJiooypaWl/nmHDh0ysbGxZsiQIRfsn6ysrIBtzZw504SHh5vjx49fMHbx4sUN9tAQt9ttJBlJplOnTub111+/7FqgIbwFhxYnKytLCQkJSktL0/jx4xUTE6OVK1fquuuuCxg3bdq0gNcrVqyQx+PRXXfdpW+//dY/9evXTzExMSooKJAkrV+/XmfOnNH06dMD3hqbMWPGJXsrLi5WWVmZZsyYofbt2wcsO39dl6u+vl5/+ctfNGbMGHXv3t0/PyUlRffff782b94sr9cbUPPQQw8FbGvw4MGqr6/XN9984583adIkGWMu++xHkj755BOtWbNGr7zyirp06aKamhrHPw9wPt6CQ4uzYMEC9ejRQxEREUpKSlLPnj0VFhb4u1RERIQ6d+4cMG/v3r2qqqpSYmJig+s9cuSIJPn/o77hhhsClickJKhDhw4X7e37twN79+59+T/QRRw9elQnT55Uz549L1h20003yefzqby8XL169fLP79KlS8C473v+4XUup+644w5J524CGT16tHr37q2YmBjl5eVd0Xpx7SKA0OL079/ffxdcY9xu9wWh5PP5lJiYqKVLlzZYk5CQELIebQoPD29wvjEmZNvIyMjQj3/8Yy1dupQAQtAIIFwzMjIytH79eg0cOFBt27ZtdFzXrl0lnTtjOv9tr6NHj17yLCIjI0OStHv3bmVlZTU67nLfjktISFB0dLRKSkouWLZnzx6FhYUpLS3tstYVaqdOnVJtba2VbaN14BoQrhm//OUvVV9fr9/+9rcXLDt79qz/luKsrCxFRkbqjTfeCDhrmD9//iW38ZOf/ETp6emaP3/+Bbcon7+u7z+TdKnbmMPDwzV8+HCtWrUq4MkDlZWVWrZsmQYNGqS4uLhL9vVDl3sb9tmzZxsM3S+++EJfffXVJc9EgYvhDAjXjNtvv11Tp05Vfn6+du7cqeHDhysyMlJ79+7VihUr9Nprr+nnP/+5EhIS9MQTTyg/P1/33HOPRo4cqeLiYn3yySeKj4+/6DbCwsK0cOFCjRo1SrfccosmT56slJQU7dmzR19//bU+/fRTSVK/fv0kSY8++qiys7MVHh6u8ePHN7jOuXPnat26dRo0aJAeeeQRRURE6K233lJtba3mzZsX1L643NuwT5w4obS0NN13333q1auX2rVrp6+++kqLFy+Wx+PRc889F9T2AYkAwjVm0aJF6tevn9566y0988wzioiIULdu3fTggw9q4MCB/nFz585VmzZttGjRIhUUFCgzM1N/+ctfdPfdd19yG9nZ2SooKNCLL76oV155RT6fTxkZGZoyZYp/zNixYzV9+nQtX75c7777rowxjQZQr1699Ne//lWzZ89Wfn6+fD6fMjMz9e67717wGaBQi46O1q9//WsVFBToz3/+s06dOqXU1FRNmDBBzz77rP8DqUAwXCaUVyYBALhMXAMCAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCKZvc5IJ/Pp0OHDik2NjaopwcDAOwyxqi6ulqpqakXPJPxfM0ugA4dOmTt2VYAgNApLy+/4Kn052t2ARQbGytJGqSRilCk5W4AAE6dVZ02a43///PGNFkALViwQC+//LIqKirUt29fvfHGG+rfv/8l675/2y1CkYpwEUAA0OL8/+frXOoySpPchPD+++9r1qxZmjNnjr788kv17dtX2dnZ/i/8AgCgSQLo1Vdf1ZQpUzR58mT96Ec/0qJFixQdHa0//vGPTbE5AEALFPIAOnPmjHbs2BHwZVxhYWHKysrSli1bLhhfW1srr9cbMAEAWr+QB9C3336r+vp6JSUlBcxPSkpSRUXFBePz8/Pl8Xj8E3fAAcC1wfoHUWfPnq2qqir/VF5ebrslAMBVEPK74OLj4xUeHq7KysqA+ZWVlUpOTr5gvNvtltvtDnUbAIBmLuRnQFFRUerXr582bNjgn+fz+bRhwwYNGDAg1JsDALRQTfI5oFmzZmnixIm69dZb1b9/f82fP181NTWaPHlyU2wOANACNUkA3XfffTp69Kief/55VVRU6JZbbtHatWsvuDEBAHDtchljjO0mzuf1euXxeDRUo3kSAgC0QGdNnQq1SlVVVYqLi2t0nPW74AAA1yYCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWBFhuwHYZ37aN7i6cOe/v0RWeh3XlP7vRMc1vu6nHNdI0p7b/+i4JtzlfD9MPTjAcU3hp7c4run2nzWOayRJW3cFVwc4wBkQAMAKAggAYEXIA+iFF16Qy+UKmG688cZQbwYA0MI1yTWgXr16af369f+zkQguNQEAAjVJMkRERCg5ObkpVg0AaCWa5BrQ3r17lZqaqu7du+uBBx7QgQMHGh1bW1srr9cbMAEAWr+QB1BmZqaWLFmitWvXauHChSorK9PgwYNVXV3d4Pj8/Hx5PB7/lJaWFuqWAADNUMgDKCcnR7/4xS/Up08fZWdna82aNTp+/Lg++OCDBsfPnj1bVVVV/qm8vDzULQEAmqEmvzugffv26tGjh/bt29fgcrfbLbfb3dRtAACamSb/HNCJEydUWlqqlJSUpt4UAKAFCXkAPfHEEyoqKtL+/fv1+eef695771V4eLgmTJgQ6k0BAFqwkL8Fd/DgQU2YMEHHjh1TQkKCBg0apK1btyohISHUmwIAtGAuY4yx3cT5vF6vPB6Phmq0IlyRttuxqubnmY5rKm91flK7dsLLjmskqUtEW8c1/2v/XY5r/tRtneManFN8xhdU3eOP5zmuif5wW1DbQutz1tSpUKtUVVWluLi4RsfxLDgAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIKHkV4lR/J+6rim8OlXHNdEu6Ic1zR339afclzTxhXc71Z1cv7PYcaBexzX/DLxvxzX3B1d5bgmWPvqah3XPDH4l45rzpYfdFyD5o+HkQIAmjUCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsiLDdwLXCF+68pjU+2frlYz9yXLNhxiDHNfVtg/vd6rvrnT+B/br/POy45s2EcY5r7v7zHx3XBGvsf011XNPt+P7QN4JWjTMgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCh5FeJan/+jfHNR88kui4Jjv6gOOanDlPOK6RpLoYl+Oa6/7joOOaiP07nNc4rjgnOYia+iBqKu/5aRBVV8+uny5xXDMmiAes+qqrHdeg9eAMCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCs4GGkV4mvpsZxzTs90xzXvJ0z1nFNfEGx4xpJ8p0+7bjmbFBbunrCExIc13x3V4bjmscf/sBxDdDacAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcByY4yef/55paSkqG3btsrKytLevXtD1S8AoJVwHEA1NTXq27evFixY0ODyefPm6fXXX9eiRYu0bds2tWvXTtnZ2TodxPUCAEDr5fgmhJycHOXk5DS4zBij+fPn69lnn9Xo0aMlSe+8846SkpL00Ucfafz48VfWLQCg1QjpNaCysjJVVFQoKyvLP8/j8SgzM1NbtmxpsKa2tlZerzdgAgC0fiENoIqKCklSUlJSwPykpCT/sh/Kz8+Xx+PxT2lpzm89BgC0PNbvgps9e7aqqqr8U3l5ue2WAABXQUgDKDk5WZJUWVkZML+ystK/7Ifcbrfi4uICJgBA6xfSAEpPT1dycrI2bNjgn+f1erVt2zYNGDAglJsCALRwju+CO3HihPbt2+d/XVZWpp07d6pjx47q0qWLZsyYoblz5+qGG25Qenq6nnvuOaWmpmrMmDGh7BsA0MI5DqDt27frjjvu8L+eNWuWJGnixIlasmSJnnzySdXU1Oihhx7S8ePHNWjQIK1du1Zt2rQJXdcAgBbPZYwxtps4n9frlcfj0VCNVoQr0nY7aKHC23uCqnt8x2bHNUPanAlqW1eDT76g6n73bV/HNduGpTiuqf/2mOMaNH9nTZ0KtUpVVVUXva5v/S44AMC1iQACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACscfx0D0BKUPdorqLohbTaGuBO7VtXEB1X3ed+oIKp4sjWc4QwIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKzgYaRAK3ZX28NB1c2d9YDjmrqYoDblWGLxWcc1bT7+ogk6wZXiDAgAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArOBhpGiV0jacDKpuxyTnNf3cQW3qqogJC665HY+/EeJOQmfOkR87rtnxMb9rN0f8rQAArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFTyMFK2S67OdQdW9NOp+xzW1STGOa6of9zqu+eyW5Y5rWqNnE7Y7rrnzwUeD2pbn3a1B1eHycAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcDySZMmyeVyBUwjRowIVb8AgFbCcQDV1NSob9++WrBgQaNjRowYocOHD/un995774qaBAC0Po5vQsjJyVFOTs5Fx7jdbiUnJwfdFACg9WuSa0CFhYVKTExUz549NW3aNB07dqzRsbW1tfJ6vQETAKD1C3kAjRgxQu+88442bNigf/7nf1ZRUZFycnJUX1/f4Pj8/Hx5PB7/lJaWFuqWAADNUMg/BzR+/Hj/n2+++Wb16dNHGRkZKiws1LBhwy4YP3v2bM2aNcv/2uv1EkIAcA1o8tuwu3fvrvj4eO3bt6/B5W63W3FxcQETAKD1a/IAOnjwoI4dO6aUlJSm3hQAoAVx/BbciRMnAs5mysrKtHPnTnXs2FEdO3bUiy++qHHjxik5OVmlpaV68skndf311ys7OzukjQMAWjbHAbR9+3bdcccd/tffX7+ZOHGiFi5cqF27dunf//3fdfz4caWmpmr48OH67W9/K7fbHbquAQAtnuMAGjp0qIwxjS7/9NNPr6ghwKb6r0sc10R87Xw7HQpcjmtGRf3Ucc3+P/VwXCNJn2QudFzTOaJtUNtyKtIV7rjmdMfgrjZ4gqrC5eJZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAi5F/JDeAyXOSJ8o2W1NY6run6y68c10jSnW/NdFzz3/csCmpbuHZxBgQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVvAwUqAVc0VGBVfXtj7EnYTOrjPOe0vcXtMEneBKcQYEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFbwMFKgFSt545ag6v572MLQNhJCMx6f7rgm+vNtTdAJrhRnQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQ8jRasUFhsbXF17T4g7adiRu9Ic19w1/TPHNf+RuMBxzTlX53fTD04kOq6J2/KN45qzjitwNXAGBACwggACAFjhKIDy8/N12223KTY2VomJiRozZoxKSkoCxpw+fVq5ubnq1KmTYmJiNG7cOFVWVoa0aQBAy+cogIqKipSbm6utW7dq3bp1qqur0/Dhw1VTU+MfM3PmTH388cdasWKFioqKdOjQIY0dOzbkjQMAWjZHNyGsXbs24PWSJUuUmJioHTt2aMiQIaqqqtK//du/admyZbrzzjslSYsXL9ZNN92krVu36p/+6Z9C1zkAoEW7omtAVVVVkqSOHTtKknbs2KG6ujplZWX5x9x4443q0qWLtmzZ0uA6amtr5fV6AyYAQOsXdAD5fD7NmDFDAwcOVO/evSVJFRUVioqKUvv27QPGJiUlqaKiosH15Ofny+Px+Ke0NOe3pwIAWp6gAyg3N1e7d+/W8uXLr6iB2bNnq6qqyj+Vl5df0foAAC1DUB9EzcvL0+rVq7Vp0yZ17tzZPz85OVlnzpzR8ePHA86CKisrlZyc3OC63G633G53MG0AAFowR2dAxhjl5eVp5cqV2rhxo9LT0wOW9+vXT5GRkdqwYYN/XklJiQ4cOKABAwaEpmMAQKvg6AwoNzdXy5Yt06pVqxQbG+u/ruPxeNS2bVt5PB796le/0qxZs9SxY0fFxcVp+vTpGjBgAHfAAQACOAqghQsXSpKGDh0aMH/x4sWaNGmSJOn3v/+9wsLCNG7cONXW1io7O1tvvvlmSJoFALQeLmOMsd3E+bxerzwej4ZqtCJckbbbuSaE9b0pqLo9uTGOa5LT/q/jmiMlCY5rJt9Z6LhGkp7q9HVQdQhOn88nOa7p8ouvQt8IQuqsqVOhVqmqqkpxcXGNjuNZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAiqG9ERfPl6tfLcU3b3x8Jalv/nfFuUHWO9bk6m2nuak2d45pIV3hQ26qsr3VcM+dQjuOazq8F1x9aB86AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKHkbaytR0jXFc8373fw1ya1FB1jU9n3xB1c08NNhxzW8S1zuuyf4813FNbGG045rqbo5LJEnps7cEUVXtuCJMO4PYDloLzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoeRtrKRH+4zXHNLw4/HNS2jv64neMaXxDPL61z/nxVvf1//uC8SFLpbacd10z78VTHNek7dzmukTGOS+KdbwW4ajgDAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArXMYE8YTDJuT1euXxeDRUoxXhirTdDgDAobOmToVapaqqKsXFxTU6jjMgAIAVBBAAwApHAZSfn6/bbrtNsbGxSkxM1JgxY1RSUhIwZujQoXK5XAHTww8H930zAIDWy1EAFRUVKTc3V1u3btW6detUV1en4cOHq6amJmDclClTdPjwYf80b968kDYNAGj5HH0j6tq1awNeL1myRImJidqxY4eGDBninx8dHa3k5OTQdAgAaJWu6BpQVVWVJKljx44B85cuXar4+Hj17t1bs2fP1smTJxtdR21trbxeb8AEAGj9HJ0Bnc/n82nGjBkaOHCgevfu7Z9///33q2vXrkpNTdWuXbv01FNPqaSkRB9++GGD68nPz9eLL74YbBsAgBYq6M8BTZs2TZ988ok2b96szp07Nzpu48aNGjZsmPbt26eMjIwLltfW1qq2ttb/2uv1Ki0tjc8BAUALdbmfAwrqDCgvL0+rV6/Wpk2bLho+kpSZmSlJjQaQ2+2W2+0Opg0AQAvmKICMMZo+fbpWrlypwsJCpaenX7Jm586dkqSUlJSgGgQAtE6OAig3N1fLli3TqlWrFBsbq4qKCkmSx+NR27ZtVVpaqmXLlmnkyJHq1KmTdu3apZkzZ2rIkCHq06dPk/wAAICWydE1IJfL1eD8xYsXa9KkSSovL9eDDz6o3bt3q6amRmlpabr33nv17LPPXvR9wPPxLDgAaNma5BrQpbIqLS1NRUVFTlYJALhG8Sw4AIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVEbYb+CFjjCTprOokY7kZAIBjZ1Un6X/+P29Mswug6upqSdJmrbHcCQDgSlRXV8vj8TS63GUuFVFXmc/n06FDhxQbGyuXyxWwzOv1Ki0tTeXl5YqLi7PUoX3sh3PYD+ewH85hP5zTHPaDMUbV1dVKTU1VWFjjV3qa3RlQWFiYOnfufNExcXFx1/QB9j32wznsh3PYD+ewH86xvR8udubzPW5CAABYQQABAKxoUQHkdrs1Z84cud1u261YxX44h/1wDvvhHPbDOS1pPzS7mxAAANeGFnUGBABoPQggAIAVBBAAwAoCCABgBQEEALCixQTQggUL1K1bN7Vp00aZmZn64osvbLd01b3wwgtyuVwB04033mi7rSa3adMmjRo1SqmpqXK5XProo48Clhtj9PzzzyslJUVt27ZVVlaW9u7da6fZJnSp/TBp0qQLjo8RI0bYabaJ5Ofn67bbblNsbKwSExM1ZswYlZSUBIw5ffq0cnNz1alTJ8XExGjcuHGqrKy01HHTuJz9MHTo0AuOh4cffthSxw1rEQH0/vvva9asWZozZ46+/PJL9e3bV9nZ2Tpy5Ijt1q66Xr166fDhw/5p8+bNtltqcjU1Nerbt68WLFjQ4PJ58+bp9ddf16JFi7Rt2za1a9dO2dnZOn369FXutGldaj9I0ogRIwKOj/fee+8qdtj0ioqKlJubq61bt2rdunWqq6vT8OHDVVNT4x8zc+ZMffzxx1qxYoWKiop06NAhjR071mLXoXc5+0GSpkyZEnA8zJs3z1LHjTAtQP/+/U1ubq7/dX19vUlNTTX5+fkWu7r65syZY/r27Wu7DaskmZUrV/pf+3w+k5ycbF5++WX/vOPHjxu3223ee+89Cx1eHT/cD8YYM3HiRDN69Ggr/dhy5MgRI8kUFRUZY8793UdGRpoVK1b4x/zjH/8wksyWLVtstdnkfrgfjDHm9ttvN4899pi9pi5Dsz8DOnPmjHbs2KGsrCz/vLCwMGVlZWnLli0WO7Nj7969Sk1NVffu3fXAAw/owIEDtluyqqysTBUVFQHHh8fjUWZm5jV5fBQWFioxMVE9e/bUtGnTdOzYMdstNamqqipJUseOHSVJO3bsUF1dXcDxcOONN6pLly6t+nj44X743tKlSxUfH6/evXtr9uzZOnnypI32GtXsnob9Q99++63q6+uVlJQUMD8pKUl79uyx1JUdmZmZWrJkiXr27KnDhw/rxRdf1ODBg7V7927Fxsbabs+KiooKSWrw+Ph+2bVixIgRGjt2rNLT01VaWqpnnnlGOTk52rJli8LDw223F3I+n08zZszQwIED1bt3b0nnjoeoqCi1b98+YGxrPh4a2g+SdP/996tr165KTU3Vrl279NRTT6mkpEQffvihxW4DNfsAwv/Iycnx/7lPnz7KzMxU165d9cEHH+hXv/qVxc7QHIwfP97/55tvvll9+vRRRkaGCgsLNWzYMIudNY3c3Fzt3r37mrgOejGN7YeHHnrI/+ebb75ZKSkpGjZsmEpLS5WRkXG122xQs38LLj4+XuHh4RfcxVJZWank5GRLXTUP7du3V48ePbRv3z7brVjz/THA8XGh7t27Kz4+vlUeH3l5eVq9erUKCgoCvj8sOTlZZ86c0fHjxwPGt9bjobH90JDMzExJalbHQ7MPoKioKPXr108bNmzwz/P5fNqwYYMGDBhgsTP7Tpw4odLSUqWkpNhuxZr09HQlJycHHB9er1fbtm275o+PgwcP6tixY63q+DDGKC8vTytXrtTGjRuVnp4esLxfv36KjIwMOB5KSkp04MCBVnU8XGo/NGTnzp2S1LyOB9t3QVyO5cuXG7fbbZYsWWL+/ve/m4ceesi0b9/eVFRU2G7tqnr88cdNYWGhKSsrM5999pnJysoy8fHx5siRI7Zba1LV1dWmuLjYFBcXG0nm1VdfNcXFxeabb74xxhjzu9/9zrRv396sWrXK7Nq1y4wePdqkp6ebU6dOWe48tC62H6qrq80TTzxhtmzZYsrKysz69evNT37yE3PDDTeY06dP2249ZKZNm2Y8Ho8pLCw0hw8f9k8nT570j3n44YdNly5dzMaNG8327dvNgAEDzIABAyx2HXqX2g/79u0zL730ktm+fbspKyszq1atMt27dzdDhgyx3HmgFhFAxhjzxhtvmC5dupioqCjTv39/s3XrVtstXXX33XefSUlJMVFRUea6664z9913n9m3b5/ttppcQUGBkXTBNHHiRGPMuVuxn3vuOZOUlGTcbrcZNmyYKSkpsdt0E7jYfjh58qQZPny4SUhIMJGRkaZr165mypQpre6XtIZ+fklm8eLF/jGnTp0yjzzyiOnQoYOJjo429957rzl8+LC9ppvApfbDgQMHzJAhQ0zHjh2N2+02119/vfnNb35jqqqq7Db+A3wfEADAimZ/DQgA0DoRQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAV/w/hgVLrpVGHsAAAAABJRU5ErkJggg==", "text/plain": [ "
" ] @@ -1725,7 +1948,7 @@ }, { "cell_type": "code", - "execution_count": 50, + "execution_count": 52, "id": "a64d19b1-ba4a-4dc7-b3a9-368dc47d0fd8", "metadata": {}, "outputs": [], @@ -1738,7 +1961,7 @@ }, { "cell_type": "code", - "execution_count": 51, + "execution_count": 53, "id": "8fa92fe4-2e04-4d82-a357-bfdfca38bd8c", "metadata": {}, "outputs": [], @@ -1763,7 +1986,7 @@ }, { "cell_type": "code", - "execution_count": 52, + "execution_count": 54, "id": "0f7ecb25-be16-40c4-bdbb-441e2f537000", "metadata": {}, "outputs": [ @@ -1780,7 +2003,7 @@ "[True]" ] }, - "execution_count": 52, + "execution_count": 54, "metadata": {}, "output_type": "execute_result" } @@ -1802,7 +2025,7 @@ " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", " else:\n", " container=client.containers.run(\n", - " \"nvcr.io/nvidia/tritonserver:23.04-py3\", \"tritonserver --model-repository=/models\",\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", " detach=True,\n", " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", " name=\"spark-triton\",\n", @@ -1830,7 +2053,7 @@ }, { "cell_type": "code", - "execution_count": 53, + "execution_count": 55, "id": "43b93753-1d52-4060-9986-f24c30a67528", "metadata": {}, "outputs": [ @@ -1840,7 +2063,7 @@ "StructType([StructField('data', ArrayType(DoubleType(), True), True)])" ] }, - "execution_count": 53, + "execution_count": 55, "metadata": {}, "output_type": "execute_result" } @@ -1852,127 +2075,131 @@ }, { "cell_type": "markdown", - "id": "b37d5026-8abc-45c3-a03c-3222c15e7d93", + "id": "036680eb-babd-4b07-8b2c-ce6e724f4e85", "metadata": {}, "source": [ - "#### Using custom predict_batch_fn" + "#### Run inference" ] }, { "cell_type": "code", - "execution_count": 54, - "id": "f5d4c2c6-d65a-4969-8ffd-8457b6dd35fb", + "execution_count": 56, + "id": "3af08bd0-3838-4769-a8de-2643db4101c6", "metadata": {}, "outputs": [], "source": [ - "def predict_batch_fn():\n", + "def triton_fn(triton_uri, model_name):\n", " import numpy as np\n", " import tritonclient.grpc as grpcclient\n", - " client = grpcclient.InferenceServerClient(\"localhost:8001\")\n", + "\n", + " np_types = {\n", + " \"BOOL\": np.dtype(np.bool_),\n", + " \"INT8\": np.dtype(np.int8),\n", + " \"INT16\": np.dtype(np.int16),\n", + " \"INT32\": np.dtype(np.int32),\n", + " \"INT64\": np.dtype(np.int64),\n", + " \"FP16\": np.dtype(np.float16),\n", + " \"FP32\": np.dtype(np.float32),\n", + " \"FP64\": np.dtype(np.float64),\n", + " \"FP64\": np.dtype(np.double),\n", + " \"BYTES\": np.dtype(object)\n", + " }\n", + "\n", + " client = grpcclient.InferenceServerClient(triton_uri)\n", + " model_meta = client.get_model_metadata(model_name)\n", + "\n", " def predict(inputs):\n", - " request = [grpcclient.InferInput(\"dense_input\", inputs.shape, \"FP32\")]\n", - " request[0].set_data_from_numpy(inputs.astype(np.float32))\n", - " response = client.infer(\"mnist_model\", inputs=request)\n", - " return response.as_numpy(\"dense_1\")\n", + " if isinstance(inputs, np.ndarray):\n", + " # single ndarray input\n", + " request = [grpcclient.InferInput(model_meta.inputs[0].name, inputs.shape, model_meta.inputs[0].datatype)]\n", + " request[0].set_data_from_numpy(inputs.astype(np_types[model_meta.inputs[0].datatype]))\n", + " else:\n", + " # dict of multiple ndarray inputs\n", + " request = [grpcclient.InferInput(i.name, inputs[i.name].shape, i.datatype) for i in model_meta.inputs]\n", + " for i in request:\n", + " i.set_data_from_numpy(inputs[i.name()].astype(np_types[i.datatype()]))\n", + "\n", + " response = client.infer(model_name, inputs=request)\n", + "\n", + " if len(model_meta.outputs) > 1:\n", + " # return dictionary of numpy arrays\n", + " return {o.name: response.as_numpy(o.name) for o in model_meta.outputs}\n", + " else:\n", + " # return single numpy array\n", + " return response.as_numpy(model_meta.outputs[0].name)\n", + "\n", " return predict" ] }, { "cell_type": "code", - "execution_count": 55, - "id": "a158490c-eaf6-4c0a-9f2f-af20df45d126", + "execution_count": 57, + "id": "6658d2a1-ef7b-4ca1-9fb6-f2ac9050f3e5", "metadata": {}, "outputs": [], "source": [ - "mnist = predict_batch_udf(predict_batch_fn,\n", - " input_tensor_shapes=[[784]],\n", - " return_type=ArrayType(FloatType()),\n", - " batch_size=8192)" + "from functools import partial\n", + "\n", + "predict = predict_batch_udf(partial(triton_fn, \"localhost:8001\", \"mnist_model\"),\n", + " return_type=ArrayType(FloatType()),\n", + " input_tensor_shapes=[[784]],\n", + " batch_size=8192)" ] }, { "cell_type": "code", - "execution_count": 56, - "id": "b7bd2504-1ed2-4dfe-a63a-7ed660dd1c80", + "execution_count": 58, + "id": "8397aa14-82fd-4351-a477-dc8e8b321fa2", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 458 ms, sys: 80.5 ms, total: 538 ms\n", - "Wall time: 2.29 s\n" + "CPU times: user 20.1 ms, sys: 3.41 ms, total: 23.5 ms\n", + "Wall time: 625 ms\n" ] } ], "source": [ "%%time\n", - "preds = df.withColumn(\"preds\", mnist(struct(\"data\"))).collect()" + "preds = df.withColumn(\"preds\", predict(struct(\"data\"))).collect()" ] }, { "cell_type": "code", - "execution_count": 57, - "id": "4b26e237-3690-4350-a5b8-a354e668e9b7", + "execution_count": 59, + "id": "82698bd9-377a-4415-8971-835487f876cc", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 455 ms, sys: 157 ms, total: 612 ms\n", - "Wall time: 1.31 s\n" + "CPU times: user 30.3 ms, sys: 8.81 ms, total: 39.2 ms\n", + "Wall time: 154 ms\n" ] } ], "source": [ "%%time\n", - "preds = df.withColumn(\"preds\", mnist(\"data\")).collect()" + "preds = df.withColumn(\"preds\", predict(\"data\")).collect()" ] }, { "cell_type": "code", - "execution_count": 58, - "id": "d6c19561-16a9-4ce4-b7df-dca9820d6173", + "execution_count": 60, + "id": "419ad7bd-fa28-49d3-b98d-db9fba5aeaef", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 874 ms, sys: 68.9 ms, total: 943 ms\n", - "Wall time: 1.53 s\n" + "CPU times: user 2.67 ms, sys: 4.2 ms, total: 6.87 ms\n", + "Wall time: 131 ms\n" ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", mnist(col(\"data\"))).collect()" - ] - }, - { - "cell_type": "markdown", - "id": "50b58a82-78de-4749-a7b9-d6f38352fc69", - "metadata": { - "tags": [] - }, - "source": [ - "#### Check predictions" - ] - }, - { - "cell_type": "code", - "execution_count": 59, - "id": "bcf123ea-c0f4-4ec7-b52f-c1e9581b83e7", - "metadata": {}, - "outputs": [ + }, { "data": { "text/html": [ @@ -2002,60 +2229,60 @@ " \n", " 0\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-9.66701889038086, -3.6094400882720947, -3.31...\n", + " [-5.7614846, -3.52228, -1.1202906, 13.053683, ...\n", " \n", " \n", " 1\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-16.348302841186523, 3.1133782863616943, 6.73...\n", + " [-3.1390061, -8.71185, 0.82955813, -4.034869, ...\n", " \n", " \n", " 2\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-21.153541564941406, -6.192654132843018, -6.6...\n", + " [-3.046528, 0.3521706, 0.6788677, 0.72303534, ...\n", " \n", " \n", " 3\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-13.790016174316406, -16.204233169555664, -8....\n", + " [-2.401024, -7.6780066, 11.145876, 1.2857256, ...\n", " \n", " \n", " 4\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-18.723485946655273, -7.8062005043029785, -8....\n", + " [-5.0012593, 3.806796, -0.8154834, -0.9550028,...\n", " \n", " \n", " 5\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [10.495272636413574, -10.992119789123535, -2.4...\n", + " [-5.0425925, -3.4815094, 1.641246, 3.608149, -...\n", " \n", " \n", " 6\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-9.437854766845703, -3.745434284210205, -3.59...\n", + " [-4.288771, 5.0072904, 0.27649477, -0.797148, ...\n", " \n", " \n", " 7\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-12.314884185791016, -2.106672525405884, -0.7...\n", + " [-2.2032878, -1.6879876, -5.874276, -0.5945335...\n", " \n", " \n", " 8\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-22.142610549926758, -2.891775369644165, -4.8...\n", + " [1.1337761, -3.1751056, -2.5246286, -5.028277,...\n", " \n", " \n", " 9\n", " [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...\n", - " [-12.939340591430664, 0.40699198842048645, 15....\n", + " [-0.92484117, -2.4703276, -5.023897, 1.46669, ...\n", " \n", " \n", "\n", "" ], "text/plain": [ - " data \n", - "0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \\\n", + " data \\\n", + "0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", "1 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", "2 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", "3 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", @@ -2067,32 +2294,33 @@ "9 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", "\n", " preds \n", - "0 [-9.66701889038086, -3.6094400882720947, -3.31... \n", - "1 [-16.348302841186523, 3.1133782863616943, 6.73... \n", - "2 [-21.153541564941406, -6.192654132843018, -6.6... \n", - "3 [-13.790016174316406, -16.204233169555664, -8.... \n", - "4 [-18.723485946655273, -7.8062005043029785, -8.... \n", - "5 [10.495272636413574, -10.992119789123535, -2.4... \n", - "6 [-9.437854766845703, -3.745434284210205, -3.59... \n", - "7 [-12.314884185791016, -2.106672525405884, -0.7... \n", - "8 [-22.142610549926758, -2.891775369644165, -4.8... \n", - "9 [-12.939340591430664, 0.40699198842048645, 15.... " + "0 [-5.7614846, -3.52228, -1.1202906, 13.053683, ... \n", + "1 [-3.1390061, -8.71185, 0.82955813, -4.034869, ... \n", + "2 [-3.046528, 0.3521706, 0.6788677, 0.72303534, ... \n", + "3 [-2.401024, -7.6780066, 11.145876, 1.2857256, ... \n", + "4 [-5.0012593, 3.806796, -0.8154834, -0.9550028,... \n", + "5 [-5.0425925, -3.4815094, 1.641246, 3.608149, -... \n", + "6 [-4.288771, 5.0072904, 0.27649477, -0.797148, ... \n", + "7 [-2.2032878, -1.6879876, -5.874276, -0.5945335... \n", + "8 [1.1337761, -3.1751056, -2.5246286, -5.028277,... \n", + "9 [-0.92484117, -2.4703276, -5.023897, 1.46669, ... " ] }, - "execution_count": 59, + "execution_count": 60, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "preds = df.withColumn(\"preds\", mnist(*df.columns)).limit(10).toPandas()\n", + "%%time\n", + "preds = df.withColumn(\"preds\", predict(col(\"data\"))).limit(10).toPandas()\n", "preds" ] }, { "cell_type": "code", - "execution_count": 60, - "id": "525f0157-6dac-4b62-a88c-4294a9b5e7f5", + "execution_count": 61, + "id": "79d90a26", "metadata": {}, "outputs": [], "source": [ @@ -2100,44 +2328,16 @@ "import numpy as np" ] }, - { - "cell_type": "code", - "execution_count": 61, - "id": "e20b2b47-f5c3-44ad-b971-184b1037c721", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "[-9.66701889038086,\n", - " -3.6094400882720947,\n", - " -3.310295581817627,\n", - " -0.3262983560562134,\n", - " -8.564104080200195,\n", - " -3.614396095275879,\n", - " -5.3909592628479,\n", - " -3.0429911613464355,\n", - " 9.717466354370117,\n", - " -5.417511940002441]" - ] - }, - "execution_count": 61, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "sample = preds.iloc[0]\n", - "sample.preds" - ] - }, { "cell_type": "code", "execution_count": 62, - "id": "ca2811fa-f9d4-4c14-b203-c92f56924d8b", + "id": "4ca495f5", "metadata": {}, "outputs": [], "source": [ + "sample = preds.iloc[0]\n", + "sample.preds\n", + "\n", "prediction = np.argmax(sample.preds)\n", "img = np.array(sample.data).reshape(28,28)" ] @@ -2145,12 +2345,12 @@ { "cell_type": "code", "execution_count": 63, - "id": "60dd6b57-85a0-4357-b46c-75bce626ecce", + "id": "a5d10903", "metadata": {}, "outputs": [ { "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAk1klEQVR4nO3dfXRU9b3v8c/kaXhKBkKeJWBAHixIrBQihwdRUkJQFk/HIz7cBegBxIAgtXpiVUS5Ny2eemhpBM+qEm1BkNMiS0tRnhJEgVMQDherKeRECRcSkCszEEiA5Hf/4DJ1IAH3OMkvCe/XWnstZu/fd/aXzYYPe/aeX1zGGCMAABpZmO0GAADXJwIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIcODGG2/U5MmT/a8LCwvlcrlUWFgYsn24XC698MILIXs/oKkigNBsFBQUyOVy+ZdWrVqpR48emjlzpioqKmy358i6deuaVci88847uv3229W+fXt17NhRd9xxh/70pz/ZbgvNXITtBgCnXnzxRaWlpamqqkrbtm3TkiVLtG7dOu3fv19t2rRp1F6GDh2qs2fPKioqylHdunXrlJ+fX2cInT17VhERTeev5uLFi/X444/r7rvv1s9//nNVVVWpoKBA99xzj/7whz9o/PjxtltEM9V0znLgO8rOztaPfvQjSdI///M/q2PHjnrllVe0du1a3X///XXWVFZWqm3btiHvJSwsTK1atQrpe4b6/b6vxYsXq3///nrvvffkcrkkSQ8//LBuuOEGvfnmmwQQgsZHcGj27rrrLklSaWmpJGny5Mlq166dSkpKNGrUKEVHR+vBBx+UJNXW1mrRokXq3bu3WrVqpcTERE2fPl3ffPNNwHsaY7RgwQJ16tRJbdq00Z133qnPPvvsin3Xdw9o586dGjVqlDp06KC2bduqb9+++tWvfuXvLz8/X5ICPlK8pK57QHv27FF2drZiYmLUrl07DR8+XDt27AgYc+kjyo8//lhz585VfHy82rZtq3Hjxun48eMBY71er7744gt5vd5rHl+fz6eEhISAHi/10bp162vWA/XhCgjNXklJiSSpY8eO/nUXLlxQVlaWBg8erH/913/1fzQ3ffp0FRQUaMqUKXr88cdVWlqq3/zmN9qzZ48+/vhjRUZGSpKef/55LViwQKNGjdKoUaP06aefasSIETp37tw1+9mwYYPuueceJScna/bs2UpKStLnn3+u999/X7Nnz9b06dN15MgRbdiwQb/73e+u+X6fffaZhgwZopiYGD311FOKjIzUa6+9pmHDhqmoqEgZGRkB42fNmqUOHTpo3rx5+vLLL7Vo0SLNnDlTq1at8o9Zs2aNpkyZomXLlgU8VFGXYcOG6T/+4z+0ePFijR49WlVVVVq8eLG8Xq9mz559zf6BehmgmVi2bJmRZDZu3GiOHz9uysrKzMqVK03Hjh1N69atzeHDh40xxkyaNMlIMv/yL/8SUP/RRx8ZSWb58uUB69evXx+w/tixYyYqKsrcfffdpra21j/umWeeMZLMpEmT/Ou2bNliJJktW7YYY4y5cOGCSUtLM126dDHffPNNwH6+/V45OTmmvr9+ksy8efP8r8eOHWuioqJMSUmJf92RI0dMdHS0GTp06BXHJzMzM2BfTzzxhAkPDzcnT568YuyyZcvq7OHbKioqzPDhw40k/xIXF2c++eSTa9YCV8NHcGh2MjMzFR8fr9TUVE2cOFHt2rXTmjVrdMMNNwSMmzFjRsDr1atXy+Px6Mc//rG+/vpr/9KvXz+1a9dOW7ZskSRt3LhR586d06xZswI+dpozZ841e9uzZ49KS0s1Z84ctW/fPmDbt9/ru6qpqdGHH36osWPHqmvXrv71ycnJeuCBB7Rt2zb5fL6AmmnTpgXsa8iQIaqpqdFXX33lXzd58mQZY6559SNJbdq0Uc+ePTVp0iStXr1ab7zxhpKTkzV+/HgdPHjQ8e8JuISP4NDs5Ofnq0ePHoqIiFBiYqJ69uypsLDA/0tFRESoU6dOAesOHDggr9erhISEOt/32LFjkuT/h7p79+4B2+Pj49WhQ4er9nbp48A+ffp899/QVRw/flxnzpxRz549r9h28803q7a2VmVlZerdu7d/fefOnQPGXer58vtc39W9996riIgIvffee/51Y8aMUffu3fWzn/0s4KM9wAkCCM3OgAED/E/B1cftdl8RSrW1tUpISNDy5cvrrImPjw9ZjzaFh4fXud4Y4/i9/vu//1vr16/Xv//7vwesj42N1eDBg/Xxxx8H1SMgEUC4jnTr1k0bN27UoEGDrvr0VpcuXSRdvGL69sdex48fv+ZVRLdu3SRJ+/fvV2ZmZr3jvuvHcfHx8WrTpo2Ki4uv2PbFF18oLCxMqamp3+m9gnHpC741NTVXbDt//rwuXLjQYPtGy8c9IFw3/umf/kk1NTV66aWXrth24cIFnTx5UtLFe0yRkZFavHhxwFXDokWLrrmP2267TWlpaVq0aJH//S759ntd+k7S5WMuFx4erhEjRmjt2rX68ssv/esrKiq0YsUKDR48WDExMdfs63Lf9THsm266SWFhYVq1alVA/4cPH9ZHH32kH/7wh473DVzCFRCuG3fccYemT5+uvLw87d27VyNGjFBkZKQOHDig1atX61e/+pX+8R//UfHx8XryySeVl5ene+65R6NGjdKePXv05z//WXFxcVfdR1hYmJYsWaLRo0fr1ltv1ZQpU5ScnKwvvvhCn332mT744ANJUr9+/SRJjz/+uLKyshQeHq6JEyfW+Z4LFizQhg0bNHjwYD322GOKiIjQa6+9purqai1cuDCoY/FdH8OOj4/Xww8/rN/+9rcaPny4xo8fr1OnTunVV1/V2bNnlZubG9T+AYkAwnVm6dKl6tevn1577TU988wzioiI0I033qiHHnpIgwYN8o9bsGCBWrVqpaVLl2rLli3KyMjQhx9+qLvvvvua+8jKytKWLVs0f/58/fKXv1Rtba26deumqVOn+seMHz9es2bN0sqVK/X73/9exph6A6h379766KOPlJubq7y8PNXW1iojI0O///3vr/gOUENYsmSJ0tPT9frrr/sDp3///nrrrbc0dOjQBt8/Wi6XCebOJAAA3xP3gAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsKLJfQ+otrZWR44cUXR0dFCzBwMA7DLG6NSpU0pJSbliTsZva3IBdOTIkQad2woA0DjKysqumJX+25pcAEVHR0uSBmuUIhRpuRsAgFMXdF7btM7/73l9GiyA8vPz9fLLL6u8vFzp6elavHixBgwYcM26Sx+7RShSES4CCACanf8/v861bqM0yEMIq1at0ty5czVv3jx9+umnSk9PV1ZWlv8HfgEA0CAB9Morr2jq1KmaMmWKfvCDH2jp0qVq06aN3njjjYbYHQCgGQp5AJ07d067d+8O+GFcYWFhyszM1Pbt268YX11dLZ/PF7AAAFq+kAfQ119/rZqaGiUmJgasT0xMVHl5+RXj8/Ly5PF4/AtPwAHA9cH6F1Fzc3Pl9Xr9S1lZme2WAACNIORPwcXFxSk8PNz/s+QvqaioUFJS0hXj3W633G53qNsAADRxIb8CioqKUr9+/bRp0yb/utraWm3atEkDBw4M9e4AAM1Ug3wPaO7cuZo0aZJ+9KMfacCAAVq0aJEqKys1ZcqUhtgdAKAZapAAuu+++3T8+HE9//zzKi8v16233qr169df8WACAOD65TLGGNtNfJvP55PH49EwjWEmBABohi6Y8yrUWnm9XsXExNQ7zvpTcACA6xMBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArIiw3QBwLeExMY5r/vZc76D29XDWZsc1z8QVO66pMbWOa768cMZxzagdjzmukaQui1yOa1zb/yuofeH6xRUQAMAKAggAYEXIA+iFF16Qy+UKWHr16hXq3QAAmrkGuQfUu3dvbdy48e87ieBWEwAgUIMkQ0REhJKSkhrirQEALUSD3AM6cOCAUlJS1LVrVz344IM6dOhQvWOrq6vl8/kCFgBAyxfyAMrIyFBBQYHWr1+vJUuWqLS0VEOGDNGpU6fqHJ+XlyePx+NfUlNTQ90SAKAJCnkAZWdn695771Xfvn2VlZWldevW6eTJk3rnnXfqHJ+bmyuv1+tfysrKQt0SAKAJavCnA9q3b68ePXro4MGDdW53u91yu90N3QYAoIlp8O8BnT59WiUlJUpOTm7oXQEAmpGQB9CTTz6poqIiffnll/rkk080btw4hYeH6/777w/1rgAAzVjIP4I7fPiw7r//fp04cULx8fEaPHiwduzYofj4+FDvCgDQjLmMMcZ2E9/m8/nk8Xg0TGMU4Yq03Q5CzAy61XHNi7973XFNf7fzyTTxd3+o7OC4piDjNsc1Nd9847gGTd8Fc16FWiuv16uYq0wmzFxwAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFg/9AOuDbXC9+7bgmmIlFV5/u6LhGkl7+248d10Suig1qX051ml73D3W8mlVdPwxqX2dqg/ghkTU1Qe0L1y+ugAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFs2GjUZ1eeoPzokXOS07WtHFeJCnxyVrHNTXFO4Lal1Nn/tTBcc2QrMeC2leU1/nM1m7fX4LaV2NwuYOY3VtSWLu2jmtqTvzfoPZ1PeIKCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsYDJSNCrPB587rll5Ot5xzVRPmeMaSTq0sqPjmt0/bJz/x9V8843jmuiVjTNRamNy9b/FcY33xTNB7Su+TaXjmpo7gtrVdYkrIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwgslI0ahqfD7HNb97aJTjmttWv+a4RpLmxm13XNP/tTmOa3pM/4vjmqYurFUrxzUl837ouGbnQ790XFMj47hGkob89qeOazqrPKh9XY+4AgIAWEEAAQCscBxAW7du1ejRo5WSkiKXy6V33303YLsxRs8//7ySk5PVunVrZWZm6sCBA6HqFwDQQjgOoMrKSqWnpys/P7/O7QsXLtSvf/1rLV26VDt37lTbtm2VlZWlqqqq790sAKDlcPwQQnZ2trKzs+vcZozRokWL9Oyzz2rMmDGSpLfeekuJiYl69913NXHixO/XLQCgxQjpPaDS0lKVl5crMzPTv87j8SgjI0Pbt9f9dFF1dbV8Pl/AAgBo+UIaQOXlFx8/TExMDFifmJjo33a5vLw8eTwe/5KamhrKlgAATZT1p+Byc3Pl9Xr9S1lZme2WAACNIKQBlJSUJEmqqKgIWF9RUeHfdjm3262YmJiABQDQ8oU0gNLS0pSUlKRNmzb51/l8Pu3cuVMDBw4M5a4AAM2c46fgTp8+rYMHD/pfl5aWau/evYqNjVXnzp01Z84cLViwQN27d1daWpqee+45paSkaOzYsaHsGwDQzDkOoF27dunOO+/0v547d64kadKkSSooKNBTTz2lyspKTZs2TSdPntTgwYO1fv16tQpinigAQMvlMsYEN0tfA/H5fPJ4PBqmMYpwRdpuB83V7X2DKlu+eonjmtog9pOxbo7jmmAmMA1mglBJKn/kNsc1907fdO1Bl3m64+eOa/JO/MBxTeHsf3BcI0nhWz4Nqu56d8GcV6HWyuv1XvW+vvWn4AAA1ycCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsYDZs4FvOjM9wXFO42PkM2idqzzquufPVnzquaTv4uOMaSdp+66qg6pzq88kkxzU3/o8Djmtqq6oc1yB4zIYNAGjSCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFhO0GgKak3br/clzTfdgMxzUHJjifwHTfzN84rglWMJOl/sM7P3Fc0/1nex3XMLFoy8EVEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYwWSkwLcYYxzXtPsqvAE6CY1gJhWVpJH/60nHNd2WbHdcU+u4Ai0JV0AAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAWTkaJFcrndQdUd/J8/dFxTfP9vgtpXY/jtN7cFVRcfxMSigFNcAQEArCCAAABWOA6grVu3avTo0UpJSZHL5dK7774bsH3y5MlyuVwBy8iRI0PVLwCghXAcQJWVlUpPT1d+fn69Y0aOHKmjR4/6l7fffvt7NQkAaHkcP4SQnZ2t7Ozsq45xu91KSkoKuikAQMvXIPeACgsLlZCQoJ49e2rGjBk6ceJEvWOrq6vl8/kCFgBAyxfyABo5cqTeeustbdq0Sb/4xS9UVFSk7Oxs1dTU1Dk+Ly9PHo/Hv6Smpoa6JQBAExTy7wFNnDjR/+tbbrlFffv2Vbdu3VRYWKjhw4dfMT43N1dz5871v/b5fIQQAFwHGvwx7K5duyouLk4HDx6sc7vb7VZMTEzAAgBo+Ro8gA4fPqwTJ04oOTm5oXcFAGhGHH8Ed/r06YCrmdLSUu3du1exsbGKjY3V/PnzNWHCBCUlJamkpERPPfWUbrrpJmVlZYW0cQBA8+Y4gHbt2qU777zT//rS/ZtJkyZpyZIl2rdvn958802dPHlSKSkpGjFihF566SW5g5ybCwDQMjkOoGHDhskYU+/2Dz744Hs1BITC8cnBTcJZfH/9X7AOpdWnOzquubdd/V9nqM+0Dp86rpGkbb0fdlxT81lxUPvC9Yu54AAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGBFyH8kNxBq4R06OK6Z9+SbDdBJ3fJO/MBxzdZHMxzX3Lt6meOaDmGtHddI0tlO0Y5roj4Lale4jnEFBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWMBkpmrzy+3o5rrm7zaag9rXprNtxzSc/7uy4JqzTBcc1jelsnPN/GqIaoA+0bFwBAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVTEaKRhUeE+O45qUnlzVAJ3XLKx3luCaq4ivHNa5O8Y5rgnHowpmg6mL/ctxxTU1Qe8L1jCsgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCyUjRqI4+1MdxzcjWhY5r/na+ynGNJLWe7XZcY6KjHdec+8UpxzXBeNvbL6i6mr+VhLgT4EpcAQEArCCAAABWOAqgvLw89e/fX9HR0UpISNDYsWNVXFwcMKaqqko5OTnq2LGj2rVrpwkTJqiioiKkTQMAmj9HAVRUVKScnBzt2LFDGzZs0Pnz5zVixAhVVlb6xzzxxBN67733tHr1ahUVFenIkSMaP358yBsHADRvjh5CWL9+fcDrgoICJSQkaPfu3Ro6dKi8Xq9ef/11rVixQnfddZckadmyZbr55pu1Y8cO3X777aHrHADQrH2ve0Ber1eSFBsbK0navXu3zp8/r8zMTP+YXr16qXPnztq+fXud71FdXS2fzxewAABavqADqLa2VnPmzNGgQYPUp8/FR2vLy8sVFRWl9u3bB4xNTExUeXl5ne+Tl5cnj8fjX1JTU4NtCQDQjAQdQDk5Odq/f79Wrlz5vRrIzc2V1+v1L2VlZd/r/QAAzUNQX0SdOXOm3n//fW3dulWdOnXyr09KStK5c+d08uTJgKugiooKJSUl1flebrdbbrfzL/8BAJo3R1dAxhjNnDlTa9as0ebNm5WWlhawvV+/foqMjNSmTZv864qLi3Xo0CENHDgwNB0DAFoER1dAOTk5WrFihdauXavo6Gj/fR2Px6PWrVvL4/HokUce0dy5cxUbG6uYmBjNmjVLAwcO5Ak4AEAARwG0ZMkSSdKwYcMC1i9btkyTJ0+WJP3bv/2bwsLCNGHCBFVXVysrK0uvvvpqSJoFALQcjgLIGHPNMa1atVJ+fr7y8/ODbgrNQ1jbto5rFsx9owE6udJKb/+g6mr++jfHNWXP/oPjmv99828c1xy6cMZxzZ9eGua4RpLaaWdQdYATzAUHALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAK4L6iaiAJJkLFxzX/LXqBsc1I1sfcFzzwf+52XGNJNVOindcs+vRRUHsyflfvbv+NNdxTY/VzGqNposrIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwgslIETRTXe24pmB5luOauTOdT0b6UfoqxzWSVJte67imouac45o7i6Y7rukxc7fjGqAp4woIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKxgMlI0qi5LP3dcM+++dMc18+P/y3GNJN3zxTjHNTUvJTiuuanwU8c1QEvDFRAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWMFkpGhUNd9847jmL7eGO64Zpdsc11x02HFFeBA1ALgCAgBYQgABAKxwFEB5eXnq37+/oqOjlZCQoLFjx6q4uDhgzLBhw+RyuQKWRx99NKRNAwCaP0cBVFRUpJycHO3YsUMbNmzQ+fPnNWLECFVWVgaMmzp1qo4ePepfFi5cGNKmAQDNn6OHENavXx/wuqCgQAkJCdq9e7eGDh3qX9+mTRslJSWFpkMAQIv0ve4Beb1eSVJsbGzA+uXLlysuLk59+vRRbm6uzpw5U+97VFdXy+fzBSwAgJYv6Mewa2trNWfOHA0aNEh9+vTxr3/ggQfUpUsXpaSkaN++fXr66adVXFysP/7xj3W+T15enubPnx9sGwCAZspljDHBFM6YMUN//vOftW3bNnXq1KnecZs3b9bw4cN18OBBdevW7Yrt1dXVqq6u9r/2+XxKTU3VMI1RhCsymNYAABZdMOdVqLXyer2KiYmpd1xQV0AzZ87U+++/r61bt141fCQpIyNDkuoNILfbLbfbHUwbAIBmzFEAGWM0a9YsrVmzRoWFhUpLS7tmzd69eyVJycnJQTUIAGiZHAVQTk6OVqxYobVr1yo6Olrl5eWSJI/Ho9atW6ukpEQrVqzQqFGj1LFjR+3bt09PPPGEhg4dqr59+zbIbwAA0Dw5ugfkcrnqXL9s2TJNnjxZZWVleuihh7R//35VVlYqNTVV48aN07PPPnvVzwG/zefzyePxcA8IAJqpBrkHdK2sSk1NVVFRkZO3BABcp5gLDgBgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgRYTtBi5njJEkXdB5yVhuBgDg2AWdl/T3f8/r0+QC6NSpU5KkbVpnuRMAwPdx6tQpeTyeere7zLUiqpHV1tbqyJEjio6OlsvlCtjm8/mUmpqqsrIyxcTEWOrQPo7DRRyHizgOF3EcLmoKx8EYo1OnTiklJUVhYfXf6WlyV0BhYWHq1KnTVcfExMRc1yfYJRyHizgOF3EcLuI4XGT7OFztyucSHkIAAFhBAAEArGhWAeR2uzVv3jy53W7brVjFcbiI43ARx+EijsNFzek4NLmHEAAA14dmdQUEAGg5CCAAgBUEEADACgIIAGAFAQQAsKLZBFB+fr5uvPFGtWrVShkZGfrP//xP2y01uhdeeEEulytg6dWrl+22GtzWrVs1evRopaSkyOVy6d133w3YbozR888/r+TkZLVu3VqZmZk6cOCAnWYb0LWOw+TJk684P0aOHGmn2QaSl5en/v37Kzo6WgkJCRo7dqyKi4sDxlRVVSknJ0cdO3ZUu3btNGHCBFVUVFjquGF8l+MwbNiwK86HRx991FLHdWsWAbRq1SrNnTtX8+bN06effqr09HRlZWXp2LFjtltrdL1799bRo0f9y7Zt22y31OAqKyuVnp6u/Pz8OrcvXLhQv/71r7V06VLt3LlTbdu2VVZWlqqqqhq504Z1reMgSSNHjgw4P95+++1G7LDhFRUVKScnRzt27NCGDRt0/vx5jRgxQpWVlf4xTzzxhN577z2tXr1aRUVFOnLkiMaPH2+x69D7LsdBkqZOnRpwPixcuNBSx/UwzcCAAQNMTk6O/3VNTY1JSUkxeXl5FrtqfPPmzTPp6em227BKklmzZo3/dW1trUlKSjIvv/yyf93JkyeN2+02b7/9toUOG8flx8EYYyZNmmTGjBljpR9bjh07ZiSZoqIiY8zFP/vIyEizevVq/5jPP//cSDLbt2+31WaDu/w4GGPMHXfcYWbPnm2vqe+gyV8BnTt3Trt371ZmZqZ/XVhYmDIzM7V9+3aLndlx4MABpaSkqGvXrnrwwQd16NAh2y1ZVVpaqvLy8oDzw+PxKCMj47o8PwoLC5WQkKCePXtqxowZOnHihO2WGpTX65UkxcbGSpJ2796t8+fPB5wPvXr1UufOnVv0+XD5cbhk+fLliouLU58+fZSbm6szZ87YaK9eTW427Mt9/fXXqqmpUWJiYsD6xMREffHFF5a6siMjI0MFBQXq2bOnjh49qvnz52vIkCHav3+/oqOjbbdnRXl5uSTVeX5c2na9GDlypMaPH6+0tDSVlJTomWeeUXZ2trZv367w8HDb7YVcbW2t5syZo0GDBqlPnz6SLp4PUVFRat++fcDYlnw+1HUcJOmBBx5Qly5dlJKSon379unpp59WcXGx/vjHP1rsNlCTDyD8XXZ2tv/Xffv2VUZGhrp06aJ33nlHjzzyiMXO0BRMnDjR/+tbbrlFffv2Vbdu3VRYWKjhw4db7Kxh5OTkaP/+/dfFfdCrqe84TJs2zf/rW265RcnJyRo+fLhKSkrUrVu3xm6zTk3+I7i4uDiFh4df8RRLRUWFkpKSLHXVNLRv3149evTQwYMHbbdizaVzgPPjSl27dlVcXFyLPD9mzpyp999/X1u2bAn4+WFJSUk6d+6cTp48GTC+pZ4P9R2HumRkZEhSkzofmnwARUVFqV+/ftq0aZN/XW1trTZt2qSBAwda7My+06dPq6SkRMnJybZbsSYtLU1JSUkB54fP59POnTuv+/Pj8OHDOnHiRIs6P4wxmjlzptasWaPNmzcrLS0tYHu/fv0UGRkZcD4UFxfr0KFDLep8uNZxqMvevXslqWmdD7afgvguVq5cadxutykoKDB//etfzbRp00z79u1NeXm57dYa1U9+8hNTWFhoSktLzccff2wyMzNNXFycOXbsmO3WGtSpU6fMnj17zJ49e4wk88orr5g9e/aYr776yhhjzM9//nPTvn17s3btWrNv3z4zZswYk5aWZs6ePWu589C62nE4deqUefLJJ8327dtNaWmp2bhxo7nttttM9+7dTVVVle3WQ2bGjBnG4/GYwsJCc/ToUf9y5swZ/5hHH33UdO7c2WzevNns2rXLDBw40AwcONBi16F3reNw8OBB8+KLL5pdu3aZ0tJSs3btWtO1a1czdOhQy50HahYBZIwxixcvNp07dzZRUVFmwIABZseOHbZbanT33XefSU5ONlFRUeaGG24w9913nzl48KDtthrcli1bjKQrlkmTJhljLj6K/dxzz5nExETjdrvN8OHDTXFxsd2mG8DVjsOZM2fMiBEjTHx8vImMjDRdunQxU6dObXH/Savr9y/JLFu2zD/m7Nmz5rHHHjMdOnQwbdq0MePGjTNHjx6113QDuNZxOHTokBk6dKiJjY01brfb3HTTTeanP/2p8Xq9dhu/DD8PCABgRZO/BwQAaJkIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCK/wcgFXDPfhuVLAAAAABJRU5ErkJggg==", + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAkfElEQVR4nO3dfXQUdZ7v8U/nqSEkaR7yLAFCFHRAcAYly/AgSiQEZUCYGUG9F7gziJiggI6KR0Udzsksrg7qIHjcHVhHEGWOyMoiDg9JGBRwwTCIM2QhJ0g4kIBcSYcAIaR/9w+uvbQkQDUdfkl4v86pc+iq37fqm6Lgk+qqrnYZY4wAALjKwmw3AAC4NhFAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAgAPdunXTpEmT/K8LCwvlcrlUWFgYsm24XC698MILIVsf0FwRQGgxlixZIpfL5Z/atGmjHj16KC8vT5WVlbbbc2TNmjUtJmTefvtt3X777UpKSpLb7VZ6eromT56s/fv3224NLVyE7QYAp1566SWlp6fr9OnT2rx5sxYuXKg1a9Zo9+7dio6Ovqq9DBkyRKdOnVJUVJSjujVr1mjBggUNhtCpU6cUEdF8/mkWFxcrPT1dP/vZz9ShQweVlZXp7bff1urVq/W3v/1NqamptltEC9V8jnLgMuXk5OjWW2+VJP36179Wp06d9Oqrr2rVqlWaMGFCgzU1NTVq165dyHsJCwtTmzZtQrrOUK/vSr355psXzBszZoxuvfVWvfPOO3r66actdIXWgLfg0OLdeeedkqSysjJJ0qRJkxQTE6PS0lKNHDlSsbGxeuCBByRJPp9P8+fPV69evdSmTRslJSVp6tSp+u677wLWaYzR3Llz1blzZ0VHR+uOO+7Q119/fcG2G7sGtG3bNo0cOVIdOnRQu3bt1KdPH7322mv+/hYsWCBJAW8pfq+ha0DFxcXKyclRXFycYmJiNGzYMG3dujVgzPdvUX722WeaNWuWEhIS1K5dO9177706evRowNiqqirt2bNHVVVVl7OLL9CtWzdJ0vHjx4OqByTOgNAKlJaWSpI6derkn3f27FllZ2dr0KBB+pd/+Rf/W3NTp07VkiVLNHnyZD366KMqKyvTH/7wBxUXF+uzzz5TZGSkJOn555/X3LlzNXLkSI0cOVJffvmlhg8frjNnzlyyn3Xr1umee+5RSkqKHnvsMSUnJ+sf//iHVq9erccee0xTp07VoUOHtG7dOv3pT3+65Pq+/vprDR48WHFxcXryyScVGRmpt956S0OHDlVRUZEyMzMDxk+fPl0dOnTQnDlztH//fs2fP195eXl6//33/WNWrlypyZMna/HixQE3VVzMsWPHVF9frwMHDuill16SJA0bNuyyaoEGGaCFWLx4sZFk1q9fb44ePWrKy8vN8uXLTadOnUzbtm3NwYMHjTHGTJw40UgyTz/9dED9X//6VyPJLF26NGD+2rVrA+YfOXLEREVFmbvvvtv4fD7/uGeeecZIMhMnTvTPKygoMJJMQUGBMcaYs2fPmvT0dNO1a1fz3XffBWzn/HXl5uaaxv75STJz5szxvx4zZoyJiooypaWl/nmHDh0ysbGxZsiQIRfsn6ysrIBtzZw504SHh5vjx49fMHbx4sUN9tAQt9ttJBlJplOnTub111+/7FqgIbwFhxYnKytLCQkJSktL0/jx4xUTE6OVK1fquuuuCxg3bdq0gNcrVqyQx+PRXXfdpW+//dY/9evXTzExMSooKJAkrV+/XmfOnNH06dMD3hqbMWPGJXsrLi5WWVmZZsyYofbt2wcsO39dl6u+vl5/+ctfNGbMGHXv3t0/PyUlRffff782b94sr9cbUPPQQw8FbGvw4MGqr6/XN9984583adIkGWMu++xHkj755BOtWbNGr7zyirp06aKamhrHPw9wPt6CQ4uzYMEC9ejRQxEREUpKSlLPnj0VFhb4u1RERIQ6d+4cMG/v3r2qqqpSYmJig+s9cuSIJPn/o77hhhsClickJKhDhw4X7e37twN79+59+T/QRRw9elQnT55Uz549L1h20003yefzqby8XL169fLP79KlS8C473v+4XUup+644w5J524CGT16tHr37q2YmBjl5eVd0Xpx7SKA0OL079/ffxdcY9xu9wWh5PP5lJiYqKVLlzZYk5CQELIebQoPD29wvjEmZNvIyMjQj3/8Yy1dupQAQtAIIFwzMjIytH79eg0cOFBt27ZtdFzXrl0lnTtjOv9tr6NHj17yLCIjI0OStHv3bmVlZTU67nLfjktISFB0dLRKSkouWLZnzx6FhYUpLS3tstYVaqdOnVJtba2VbaN14BoQrhm//OUvVV9fr9/+9rcXLDt79qz/luKsrCxFRkbqjTfeCDhrmD9//iW38ZOf/ETp6emaP3/+Bbcon7+u7z+TdKnbmMPDwzV8+HCtWrUq4MkDlZWVWrZsmQYNGqS4uLhL9vVDl3sb9tmzZxsM3S+++EJfffXVJc9EgYvhDAjXjNtvv11Tp05Vfn6+du7cqeHDhysyMlJ79+7VihUr9Nprr+nnP/+5EhIS9MQTTyg/P1/33HOPRo4cqeLiYn3yySeKj4+/6DbCwsK0cOFCjRo1SrfccosmT56slJQU7dmzR19//bU+/fRTSVK/fv0kSY8++qiys7MVHh6u8ePHN7jOuXPnat26dRo0aJAeeeQRRURE6K233lJtba3mzZsX1L643NuwT5w4obS0NN13333q1auX2rVrp6+++kqLFy+Wx+PRc889F9T2AYkAwjVm0aJF6tevn9566y0988wzioiIULdu3fTggw9q4MCB/nFz585VmzZttGjRIhUUFCgzM1N/+ctfdPfdd19yG9nZ2SooKNCLL76oV155RT6fTxkZGZoyZYp/zNixYzV9+nQtX75c7777rowxjQZQr1699Ne//lWzZ89Wfn6+fD6fMjMz9e67717wGaBQi46O1q9//WsVFBToz3/+s06dOqXU1FRNmDBBzz77rP8DqUAwXCaUVyYBALhMXAMCAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCKZvc5IJ/Pp0OHDik2NjaopwcDAOwyxqi6ulqpqakXPJPxfM0ugA4dOmTt2VYAgNApLy+/4Kn052t2ARQbGytJGqSRilCk5W4AAE6dVZ02a43///PGNFkALViwQC+//LIqKirUt29fvfHGG+rfv/8l675/2y1CkYpwEUAA0OL8/+frXOoySpPchPD+++9r1qxZmjNnjr788kv17dtX2dnZ/i/8AgCgSQLo1Vdf1ZQpUzR58mT96Ec/0qJFixQdHa0//vGPTbE5AEALFPIAOnPmjHbs2BHwZVxhYWHKysrSli1bLhhfW1srr9cbMAEAWr+QB9C3336r+vp6JSUlBcxPSkpSRUXFBePz8/Pl8Xj8E3fAAcC1wfoHUWfPnq2qqir/VF5ebrslAMBVEPK74OLj4xUeHq7KysqA+ZWVlUpOTr5gvNvtltvtDnUbAIBmLuRnQFFRUerXr582bNjgn+fz+bRhwwYNGDAg1JsDALRQTfI5oFmzZmnixIm69dZb1b9/f82fP181NTWaPHlyU2wOANACNUkA3XfffTp69Kief/55VVRU6JZbbtHatWsvuDEBAHDtchljjO0mzuf1euXxeDRUo3kSAgC0QGdNnQq1SlVVVYqLi2t0nPW74AAA1yYCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWBFhuwHYZ37aN7i6cOe/v0RWeh3XlP7vRMc1vu6nHNdI0p7b/+i4JtzlfD9MPTjAcU3hp7c4run2nzWOayRJW3cFVwc4wBkQAMAKAggAYEXIA+iFF16Qy+UKmG688cZQbwYA0MI1yTWgXr16af369f+zkQguNQEAAjVJMkRERCg5ObkpVg0AaCWa5BrQ3r17lZqaqu7du+uBBx7QgQMHGh1bW1srr9cbMAEAWr+QB1BmZqaWLFmitWvXauHChSorK9PgwYNVXV3d4Pj8/Hx5PB7/lJaWFuqWAADNUMgDKCcnR7/4xS/Up08fZWdna82aNTp+/Lg++OCDBsfPnj1bVVVV/qm8vDzULQEAmqEmvzugffv26tGjh/bt29fgcrfbLbfb3dRtAACamSb/HNCJEydUWlqqlJSUpt4UAKAFCXkAPfHEEyoqKtL+/fv1+eef695771V4eLgmTJgQ6k0BAFqwkL8Fd/DgQU2YMEHHjh1TQkKCBg0apK1btyohISHUmwIAtGAuY4yx3cT5vF6vPB6Phmq0IlyRttuxqubnmY5rKm91flK7dsLLjmskqUtEW8c1/2v/XY5r/tRtneManFN8xhdU3eOP5zmuif5wW1DbQutz1tSpUKtUVVWluLi4RsfxLDgAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIKHkV4lR/J+6rim8OlXHNdEu6Ic1zR339afclzTxhXc71Z1cv7PYcaBexzX/DLxvxzX3B1d5bgmWPvqah3XPDH4l45rzpYfdFyD5o+HkQIAmjUCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsiLDdwLXCF+68pjU+2frlYz9yXLNhxiDHNfVtg/vd6rvrnT+B/br/POy45s2EcY5r7v7zHx3XBGvsf011XNPt+P7QN4JWjTMgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCh5FeJan/+jfHNR88kui4Jjv6gOOanDlPOK6RpLoYl+Oa6/7joOOaiP07nNc4rjgnOYia+iBqKu/5aRBVV8+uny5xXDMmiAes+qqrHdeg9eAMCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCs4GGkV4mvpsZxzTs90xzXvJ0z1nFNfEGx4xpJ8p0+7bjmbFBbunrCExIc13x3V4bjmscf/sBxDdDacAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcByY4yef/55paSkqG3btsrKytLevXtD1S8AoJVwHEA1NTXq27evFixY0ODyefPm6fXXX9eiRYu0bds2tWvXTtnZ2TodxPUCAEDr5fgmhJycHOXk5DS4zBij+fPn69lnn9Xo0aMlSe+8846SkpL00Ucfafz48VfWLQCg1QjpNaCysjJVVFQoKyvLP8/j8SgzM1NbtmxpsKa2tlZerzdgAgC0fiENoIqKCklSUlJSwPykpCT/sh/Kz8+Xx+PxT2lpzm89BgC0PNbvgps9e7aqqqr8U3l5ue2WAABXQUgDKDk5WZJUWVkZML+ystK/7Ifcbrfi4uICJgBA6xfSAEpPT1dycrI2bNjgn+f1erVt2zYNGDAglJsCALRwju+CO3HihPbt2+d/XVZWpp07d6pjx47q0qWLZsyYoblz5+qGG25Qenq6nnvuOaWmpmrMmDGh7BsA0MI5DqDt27frjjvu8L+eNWuWJGnixIlasmSJnnzySdXU1Oihhx7S8ePHNWjQIK1du1Zt2rQJXdcAgBbPZYwxtps4n9frlcfj0VCNVoQr0nY7aKHC23uCqnt8x2bHNUPanAlqW1eDT76g6n73bV/HNduGpTiuqf/2mOMaNH9nTZ0KtUpVVVUXva5v/S44AMC1iQACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACscfx0D0BKUPdorqLohbTaGuBO7VtXEB1X3ed+oIKp4sjWc4QwIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKzgYaRAK3ZX28NB1c2d9YDjmrqYoDblWGLxWcc1bT7+ogk6wZXiDAgAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArOBhpGiV0jacDKpuxyTnNf3cQW3qqogJC665HY+/EeJOQmfOkR87rtnxMb9rN0f8rQAArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFTyMFK2S67OdQdW9NOp+xzW1STGOa6of9zqu+eyW5Y5rWqNnE7Y7rrnzwUeD2pbn3a1B1eHycAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcDySZMmyeVyBUwjRowIVb8AgFbCcQDV1NSob9++WrBgQaNjRowYocOHD/un995774qaBAC0Po5vQsjJyVFOTs5Fx7jdbiUnJwfdFACg9WuSa0CFhYVKTExUz549NW3aNB07dqzRsbW1tfJ6vQETAKD1C3kAjRgxQu+88442bNigf/7nf1ZRUZFycnJUX1/f4Pj8/Hx5PB7/lJaWFuqWAADNUMg/BzR+/Hj/n2+++Wb16dNHGRkZKiws1LBhwy4YP3v2bM2aNcv/2uv1EkIAcA1o8tuwu3fvrvj4eO3bt6/B5W63W3FxcQETAKD1a/IAOnjwoI4dO6aUlJSm3hQAoAVx/BbciRMnAs5mysrKtHPnTnXs2FEdO3bUiy++qHHjxik5OVmlpaV68skndf311ys7OzukjQMAWjbHAbR9+3bdcccd/tffX7+ZOHGiFi5cqF27dunf//3fdfz4caWmpmr48OH67W9/K7fbHbquAQAtnuMAGjp0qIwxjS7/9NNPr6ghwKb6r0sc10R87Xw7HQpcjmtGRf3Ucc3+P/VwXCNJn2QudFzTOaJtUNtyKtIV7rjmdMfgrjZ4gqrC5eJZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAi5F/JDeAyXOSJ8o2W1NY6run6y68c10jSnW/NdFzz3/csCmpbuHZxBgQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVvAwUqAVc0VGBVfXtj7EnYTOrjPOe0vcXtMEneBKcQYEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFbwMFKgFSt545ag6v572MLQNhJCMx6f7rgm+vNtTdAJrhRnQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQ8jRasUFhsbXF17T4g7adiRu9Ic19w1/TPHNf+RuMBxzTlX53fTD04kOq6J2/KN45qzjitwNXAGBACwggACAFjhKIDy8/N12223KTY2VomJiRozZoxKSkoCxpw+fVq5ubnq1KmTYmJiNG7cOFVWVoa0aQBAy+cogIqKipSbm6utW7dq3bp1qqur0/Dhw1VTU+MfM3PmTH388cdasWKFioqKdOjQIY0dOzbkjQMAWjZHNyGsXbs24PWSJUuUmJioHTt2aMiQIaqqqtK//du/admyZbrzzjslSYsXL9ZNN92krVu36p/+6Z9C1zkAoEW7omtAVVVVkqSOHTtKknbs2KG6ujplZWX5x9x4443q0qWLtmzZ0uA6amtr5fV6AyYAQOsXdAD5fD7NmDFDAwcOVO/evSVJFRUVioqKUvv27QPGJiUlqaKiosH15Ofny+Px+Ke0NOe3pwIAWp6gAyg3N1e7d+/W8uXLr6iB2bNnq6qqyj+Vl5df0foAAC1DUB9EzcvL0+rVq7Vp0yZ17tzZPz85OVlnzpzR8ePHA86CKisrlZyc3OC63G633G53MG0AAFowR2dAxhjl5eVp5cqV2rhxo9LT0wOW9+vXT5GRkdqwYYN/XklJiQ4cOKABAwaEpmMAQKvg6AwoNzdXy5Yt06pVqxQbG+u/ruPxeNS2bVt5PB796le/0qxZs9SxY0fFxcVp+vTpGjBgAHfAAQACOAqghQsXSpKGDh0aMH/x4sWaNGmSJOn3v/+9wsLCNG7cONXW1io7O1tvvvlmSJoFALQeLmOMsd3E+bxerzwej4ZqtCJckbbbuSaE9b0pqLo9uTGOa5LT/q/jmiMlCY5rJt9Z6LhGkp7q9HVQdQhOn88nOa7p8ouvQt8IQuqsqVOhVqmqqkpxcXGNjuNZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAiqG9ERfPl6tfLcU3b3x8Jalv/nfFuUHWO9bk6m2nuak2d45pIV3hQ26qsr3VcM+dQjuOazq8F1x9aB86AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKHkbaytR0jXFc8373fw1ya1FB1jU9n3xB1c08NNhxzW8S1zuuyf4813FNbGG045rqbo5LJEnps7cEUVXtuCJMO4PYDloLzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoeRtrKRH+4zXHNLw4/HNS2jv64neMaXxDPL61z/nxVvf1//uC8SFLpbacd10z78VTHNek7dzmukTGOS+KdbwW4ajgDAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArXMYE8YTDJuT1euXxeDRUoxXhirTdDgDAobOmToVapaqqKsXFxTU6jjMgAIAVBBAAwApHAZSfn6/bbrtNsbGxSkxM1JgxY1RSUhIwZujQoXK5XAHTww8H930zAIDWy1EAFRUVKTc3V1u3btW6detUV1en4cOHq6amJmDclClTdPjwYf80b968kDYNAGj5HH0j6tq1awNeL1myRImJidqxY4eGDBninx8dHa3k5OTQdAgAaJWu6BpQVVWVJKljx44B85cuXar4+Hj17t1bs2fP1smTJxtdR21trbxeb8AEAGj9HJ0Bnc/n82nGjBkaOHCgevfu7Z9///33q2vXrkpNTdWuXbv01FNPqaSkRB9++GGD68nPz9eLL74YbBsAgBYq6M8BTZs2TZ988ok2b96szp07Nzpu48aNGjZsmPbt26eMjIwLltfW1qq2ttb/2uv1Ki0tjc8BAUALdbmfAwrqDCgvL0+rV6/Wpk2bLho+kpSZmSlJjQaQ2+2W2+0Opg0AQAvmKICMMZo+fbpWrlypwsJCpaenX7Jm586dkqSUlJSgGgQAtE6OAig3N1fLli3TqlWrFBsbq4qKCkmSx+NR27ZtVVpaqmXLlmnkyJHq1KmTdu3apZkzZ2rIkCHq06dPk/wAAICWydE1IJfL1eD8xYsXa9KkSSovL9eDDz6o3bt3q6amRmlpabr33nv17LPPXvR9wPPxLDgAaNma5BrQpbIqLS1NRUVFTlYJALhG8Sw4AIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVEbYb+CFjjCTprOokY7kZAIBjZ1Un6X/+P29Mswug6upqSdJmrbHcCQDgSlRXV8vj8TS63GUuFVFXmc/n06FDhxQbGyuXyxWwzOv1Ki0tTeXl5YqLi7PUoX3sh3PYD+ewH85hP5zTHPaDMUbV1dVKTU1VWFjjV3qa3RlQWFiYOnfufNExcXFx1/QB9j32wznsh3PYD+ewH86xvR8udubzPW5CAABYQQABAKxoUQHkdrs1Z84cud1u261YxX44h/1wDvvhHPbDOS1pPzS7mxAAANeGFnUGBABoPQggAIAVBBAAwAoCCABgBQEEALCixQTQggUL1K1bN7Vp00aZmZn64osvbLd01b3wwgtyuVwB04033mi7rSa3adMmjRo1SqmpqXK5XProo48Clhtj9PzzzyslJUVt27ZVVlaW9u7da6fZJnSp/TBp0qQLjo8RI0bYabaJ5Ofn67bbblNsbKwSExM1ZswYlZSUBIw5ffq0cnNz1alTJ8XExGjcuHGqrKy01HHTuJz9MHTo0AuOh4cffthSxw1rEQH0/vvva9asWZozZ46+/PJL9e3bV9nZ2Tpy5Ijt1q66Xr166fDhw/5p8+bNtltqcjU1Nerbt68WLFjQ4PJ58+bp9ddf16JFi7Rt2za1a9dO2dnZOn369FXutGldaj9I0ogRIwKOj/fee+8qdtj0ioqKlJubq61bt2rdunWqq6vT8OHDVVNT4x8zc+ZMffzxx1qxYoWKiop06NAhjR071mLXoXc5+0GSpkyZEnA8zJs3z1LHjTAtQP/+/U1ubq7/dX19vUlNTTX5+fkWu7r65syZY/r27Wu7DaskmZUrV/pf+3w+k5ycbF5++WX/vOPHjxu3223ee+89Cx1eHT/cD8YYM3HiRDN69Ggr/dhy5MgRI8kUFRUZY8793UdGRpoVK1b4x/zjH/8wksyWLVtstdnkfrgfjDHm9ttvN4899pi9pi5Dsz8DOnPmjHbs2KGsrCz/vLCwMGVlZWnLli0WO7Nj7969Sk1NVffu3fXAAw/owIEDtluyqqysTBUVFQHHh8fjUWZm5jV5fBQWFioxMVE9e/bUtGnTdOzYMdstNamqqipJUseOHSVJO3bsUF1dXcDxcOONN6pLly6t+nj44X743tKlSxUfH6/evXtr9uzZOnnypI32GtXsnob9Q99++63q6+uVlJQUMD8pKUl79uyx1JUdmZmZWrJkiXr27KnDhw/rxRdf1ODBg7V7927Fxsbabs+KiooKSWrw+Ph+2bVixIgRGjt2rNLT01VaWqpnnnlGOTk52rJli8LDw223F3I+n08zZszQwIED1bt3b0nnjoeoqCi1b98+YGxrPh4a2g+SdP/996tr165KTU3Vrl279NRTT6mkpEQffvihxW4DNfsAwv/Iycnx/7lPnz7KzMxU165d9cEHH+hXv/qVxc7QHIwfP97/55tvvll9+vRRRkaGCgsLNWzYMIudNY3c3Fzt3r37mrgOejGN7YeHHnrI/+ebb75ZKSkpGjZsmEpLS5WRkXG122xQs38LLj4+XuHh4RfcxVJZWank5GRLXTUP7du3V48ePbRv3z7brVjz/THA8XGh7t27Kz4+vlUeH3l5eVq9erUKCgoCvj8sOTlZZ86c0fHjxwPGt9bjobH90JDMzExJalbHQ7MPoKioKPXr108bNmzwz/P5fNqwYYMGDBhgsTP7Tpw4odLSUqWkpNhuxZr09HQlJycHHB9er1fbtm275o+PgwcP6tixY63q+DDGKC8vTytXrtTGjRuVnp4esLxfv36KjIwMOB5KSkp04MCBVnU8XGo/NGTnzp2S1LyOB9t3QVyO5cuXG7fbbZYsWWL+/ve/m4ceesi0b9/eVFRU2G7tqnr88cdNYWGhKSsrM5999pnJysoy8fHx5siRI7Zba1LV1dWmuLjYFBcXG0nm1VdfNcXFxeabb74xxhjzu9/9zrRv396sWrXK7Nq1y4wePdqkp6ebU6dOWe48tC62H6qrq80TTzxhtmzZYsrKysz69evNT37yE3PDDTeY06dP2249ZKZNm2Y8Ho8pLCw0hw8f9k8nT570j3n44YdNly5dzMaNG8327dvNgAEDzIABAyx2HXqX2g/79u0zL730ktm+fbspKyszq1atMt27dzdDhgyx3HmgFhFAxhjzxhtvmC5dupioqCjTv39/s3XrVtstXXX33XefSUlJMVFRUea6664z9913n9m3b5/ttppcQUGBkXTBNHHiRGPMuVuxn3vuOZOUlGTcbrcZNmyYKSkpsdt0E7jYfjh58qQZPny4SUhIMJGRkaZr165mypQpre6XtIZ+fklm8eLF/jGnTp0yjzzyiOnQoYOJjo429957rzl8+LC9ppvApfbDgQMHzJAhQ0zHjh2N2+02119/vfnNb35jqqqq7Db+A3wfEADAimZ/DQgA0DoRQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAV/w/hgVLrpVGHsAAAAABJRU5ErkJggg==", "text/plain": [ "
" ] @@ -2166,146 +2366,6 @@ "plt.show()" ] }, - { - "cell_type": "markdown", - "id": "036680eb-babd-4b07-8b2c-ce6e724f4e85", - "metadata": {}, - "source": [ - "#### Using generic Triton function" - ] - }, - { - "cell_type": "code", - "execution_count": 64, - "id": "3af08bd0-3838-4769-a8de-2643db4101c6", - "metadata": {}, - "outputs": [], - "source": [ - "def triton_fn(triton_uri, model_name):\n", - " import numpy as np\n", - " import tritonclient.grpc as grpcclient\n", - "\n", - " np_types = {\n", - " \"BOOL\": np.dtype(np.bool8),\n", - " \"INT8\": np.dtype(np.int8),\n", - " \"INT16\": np.dtype(np.int16),\n", - " \"INT32\": np.dtype(np.int32),\n", - " \"INT64\": np.dtype(np.int64),\n", - " \"FP16\": np.dtype(np.float16),\n", - " \"FP32\": np.dtype(np.float32),\n", - " \"FP64\": np.dtype(np.float64),\n", - " \"FP64\": np.dtype(np.double),\n", - " \"BYTES\": np.dtype(object)\n", - " }\n", - "\n", - " client = grpcclient.InferenceServerClient(triton_uri)\n", - " model_meta = client.get_model_metadata(model_name)\n", - "\n", - " def predict(inputs):\n", - " if isinstance(inputs, np.ndarray):\n", - " # single ndarray input\n", - " request = [grpcclient.InferInput(model_meta.inputs[0].name, inputs.shape, model_meta.inputs[0].datatype)]\n", - " request[0].set_data_from_numpy(inputs.astype(np_types[model_meta.inputs[0].datatype]))\n", - " else:\n", - " # dict of multiple ndarray inputs\n", - " request = [grpcclient.InferInput(i.name, inputs[i.name].shape, i.datatype) for i in model_meta.inputs]\n", - " for i in request:\n", - " i.set_data_from_numpy(inputs[i.name()].astype(np_types[i.datatype()]))\n", - "\n", - " response = client.infer(model_name, inputs=request)\n", - "\n", - " if len(model_meta.outputs) > 1:\n", - " # return dictionary of numpy arrays\n", - " return {o.name: response.as_numpy(o.name) for o in model_meta.outputs}\n", - " else:\n", - " # return single numpy array\n", - " return response.as_numpy(model_meta.outputs[0].name)\n", - "\n", - " return predict" - ] - }, - { - "cell_type": "code", - "execution_count": 65, - "id": "6658d2a1-ef7b-4ca1-9fb6-f2ac9050f3e5", - "metadata": {}, - "outputs": [], - "source": [ - "from functools import partial\n", - "\n", - "predict = predict_batch_udf(partial(triton_fn, \"localhost:8001\", \"mnist_model\"),\n", - " return_type=ArrayType(FloatType()),\n", - " input_tensor_shapes=[[784]],\n", - " batch_size=8192)" - ] - }, - { - "cell_type": "code", - "execution_count": 66, - "id": "8397aa14-82fd-4351-a477-dc8e8b321fa2", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 410 ms, sys: 104 ms, total: 514 ms\n", - "Wall time: 1.65 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", predict(struct(\"data\"))).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 67, - "id": "82698bd9-377a-4415-8971-835487f876cc", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 495 ms, sys: 81.4 ms, total: 576 ms\n", - "Wall time: 1.21 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", predict(\"data\")).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 68, - "id": "419ad7bd-fa28-49d3-b98d-db9fba5aeaef", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 858 ms, sys: 109 ms, total: 967 ms\n", - "Wall time: 1.58 s\n" - ] - } - ], - "source": [ - "%%time\n", - "preds = df.withColumn(\"preds\", predict(col(\"data\"))).collect()" - ] - }, { "cell_type": "markdown", "id": "6377f41a-5654-410b-8bad-d392e9dce7b8", @@ -2318,7 +2378,7 @@ }, { "cell_type": "code", - "execution_count": 69, + "execution_count": 64, "id": "9c9fd967-5cd9-4265-add9-db5c1ccf9893", "metadata": {}, "outputs": [ @@ -2335,7 +2395,7 @@ "[True]" ] }, - "execution_count": 69, + "execution_count": 64, "metadata": {}, "output_type": "execute_result" } @@ -2359,7 +2419,7 @@ }, { "cell_type": "code", - "execution_count": 70, + "execution_count": 65, "id": "f612dc0b-538f-4ecf-81f7-ef6b58c493ab", "metadata": {}, "outputs": [], @@ -2378,7 +2438,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "spark-dl-tf", "language": "python", "name": "python3" }, @@ -2392,7 +2452,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.10" + "version": "3.11.9" } }, "nbformat": 4, diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/image_classification_tf.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/image_classification_tf.ipynb new file mode 100644 index 000000000..5add2686b --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/image_classification_tf.ipynb @@ -0,0 +1,2460 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "52d55e3f", + "metadata": {}, + "source": [ + "# Pyspark TensorFlow Inference\n", + "\n", + "## Image classification\n", + "Based on: https://www.tensorflow.org/tutorials/keras/save_and_load" + ] + }, + { + "cell_type": "markdown", + "id": "5233632d", + "metadata": {}, + "source": [ + "### Using TensorFlow\n", + "Note that cuFFT/cuDNN/cuBLAS registration errors are expected with `tf=2.17.0` and will not affect behavior, as noted in [this issue.](https://github.com/tensorflow/tensorflow/issues/62075) \n", + "This notebook does not demonstrate inference with TensorRT, as [TF-TRT](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#tensorrt-10) does not yet support `tf=2.17.0`. See the `pytorch` notebooks for TensorRT demos." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "c8b28f02", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:40:20.324462: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", + "2024-10-03 17:40:20.331437: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", + "2024-10-03 17:40:20.339109: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", + "2024-10-03 17:40:20.341362: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n", + "2024-10-03 17:40:20.347337: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", + "To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", + "2024-10-03 17:40:20.672391: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2.17.0\n" + ] + } + ], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np\n", + "import subprocess\n", + "import tensorflow as tf\n", + "import os\n", + "\n", + "from tensorflow import keras\n", + "\n", + "print(tf.version.VERSION)" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "e2e67086", + "metadata": {}, + "outputs": [], + "source": [ + "# Enable GPU memory growth\n", + "gpus = tf.config.experimental.list_physical_devices('GPU')\n", + "if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "markdown", + "id": "7e0c7ad6", + "metadata": {}, + "source": [ + "### Load and preprocess dataset" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "5b007f7c", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m11490434/11490434\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 0us/step\n" + ] + }, + { + "data": { + "text/plain": [ + "((60000, 28, 28), (10000, 28, 28))" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# load dataset as numpy arrays\n", + "(train_images, train_labels), (test_images, test_labels) = tf.keras.datasets.mnist.load_data()\n", + "train_images.shape, test_images.shape" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "7b7cedd1", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "((1000, 784), (1000, 784))" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "train_labels = train_labels[:1000]\n", + "test_labels = test_labels[:1000]\n", + "\n", + "train_images = train_images[:1000].reshape(-1, 28 * 28) / 255.0\n", + "test_images = test_images[:1000].reshape(-1, 28 * 28) / 255.0\n", + "\n", + "train_images.shape, test_images.shape" + ] + }, + { + "cell_type": "markdown", + "id": "867a4403", + "metadata": {}, + "source": [ + "### Define a model" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "746d94db", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/rishic/anaconda3/envs/spark-dl-tf/lib/python3.11/site-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n", + " super().__init__(activity_regularizer=activity_regularizer, **kwargs)\n", + "2024-10-03 17:40:21.624052: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2021] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 45743 MB memory: -> device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:01:00.0, compute capability: 8.6\n" + ] + }, + { + "data": { + "text/html": [ + "
Model: \"sequential\"\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1mModel: \"sequential\"\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
+       "┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃\n",
+       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
+       "│ dense (Dense)                   │ (None, 512)            │       401,920 │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ dropout (Dropout)               │ (None, 512)            │             0 │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ dense_1 (Dense)                 │ (None, 10)             │         5,130 │\n",
+       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
+       "
\n" + ], + "text/plain": [ + "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n", + "┃\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0m┃\n", + "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n", + "│ dense (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m512\u001b[0m) │ \u001b[38;5;34m401,920\u001b[0m │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ dropout (\u001b[38;5;33mDropout\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m512\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ dense_1 (\u001b[38;5;33mDense\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m10\u001b[0m) │ \u001b[38;5;34m5,130\u001b[0m │\n", + "└─────────────────────────────────┴────────────────────────┴───────────────┘\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Total params: 407,050 (1.55 MB)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m407,050\u001b[0m (1.55 MB)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Trainable params: 407,050 (1.55 MB)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m407,050\u001b[0m (1.55 MB)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Non-trainable params: 0 (0.00 B)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# Define a simple sequential model\n", + "def create_model():\n", + " model = tf.keras.Sequential([\n", + " keras.layers.Dense(512, activation='relu', input_shape=(784,)),\n", + " keras.layers.Dropout(0.2),\n", + " keras.layers.Dense(10)\n", + " ])\n", + "\n", + " model.compile(optimizer='adam',\n", + " loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n", + " metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])\n", + "\n", + " return model\n", + "\n", + "# Create a basic model instance\n", + "model = create_model()\n", + "\n", + "# Display the model's architecture\n", + "model.summary()" + ] + }, + { + "cell_type": "markdown", + "id": "605d082a", + "metadata": {}, + "source": [ + "### Save checkpoints during training" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "244746be", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/10\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", + "I0000 00:00:1727977222.161202 1835280 service.cc:146] XLA service 0x7ec778008e00 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n", + "I0000 00:00:1727977222.161216 1835280 service.cc:154] StreamExecutor device (0): NVIDIA RTX A6000, Compute Capability 8.6\n", + "2024-10-03 17:40:22.168848: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n", + "2024-10-03 17:40:22.206298: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 8907\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m24s\u001b[0m 778ms/step - loss: 2.3278 - sparse_categorical_accuracy: 0.1250" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "I0000 00:00:1727977222.715572 1835280 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 17ms/step - loss: 1.5867 - sparse_categorical_accuracy: 0.5096 " + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:40:23.780912: I external/local_xla/xla/stream_executor/cuda/cuda_asm_compiler.cc:393] ptxas warning : Registers are spilled to local memory in function 'gemm_fusion_dot_33', 4 bytes spill stores, 4 bytes spill loads\n", + "\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Epoch 1: val_sparse_categorical_accuracy improved from -inf to 0.78700, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 44ms/step - loss: 1.5733 - sparse_categorical_accuracy: 0.5144 - val_loss: 0.7061 - val_sparse_categorical_accuracy: 0.7870\n", + "Epoch 2/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 13ms/step - loss: 0.5514 - sparse_categorical_accuracy: 0.8438\n", + "Epoch 2: val_sparse_categorical_accuracy improved from 0.78700 to 0.83700, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step - loss: 0.4276 - sparse_categorical_accuracy: 0.8935 - val_loss: 0.5268 - val_sparse_categorical_accuracy: 0.8370\n", + "Epoch 3/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 20ms/step - loss: 0.1458 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 3: val_sparse_categorical_accuracy improved from 0.83700 to 0.85600, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.2721 - sparse_categorical_accuracy: 0.9236 - val_loss: 0.4716 - val_sparse_categorical_accuracy: 0.8560\n", + "Epoch 4/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 13ms/step - loss: 0.2223 - sparse_categorical_accuracy: 0.9375\n", + "Epoch 4: val_sparse_categorical_accuracy did not improve from 0.85600\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.2159 - sparse_categorical_accuracy: 0.9547 - val_loss: 0.4682 - val_sparse_categorical_accuracy: 0.8540\n", + "Epoch 5/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 16ms/step - loss: 0.1483 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 5: val_sparse_categorical_accuracy improved from 0.85600 to 0.86900, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 2ms/step - loss: 0.1457 - sparse_categorical_accuracy: 0.9716 - val_loss: 0.4285 - val_sparse_categorical_accuracy: 0.8690\n", + "Epoch 6/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 13ms/step - loss: 0.0836 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 6: val_sparse_categorical_accuracy did not improve from 0.86900\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.1292 - sparse_categorical_accuracy: 0.9712 - val_loss: 0.4551 - val_sparse_categorical_accuracy: 0.8580\n", + "Epoch 7/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 12ms/step - loss: 0.0920 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 7: val_sparse_categorical_accuracy did not improve from 0.86900\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.0974 - sparse_categorical_accuracy: 0.9822 - val_loss: 0.4016 - val_sparse_categorical_accuracy: 0.8670\n", + "Epoch 8/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 12ms/step - loss: 0.0993 - sparse_categorical_accuracy: 0.9688\n", + "Epoch 8: val_sparse_categorical_accuracy did not improve from 0.86900\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.0702 - sparse_categorical_accuracy: 0.9920 - val_loss: 0.3999 - val_sparse_categorical_accuracy: 0.8650\n", + "Epoch 9/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 12ms/step - loss: 0.0599 - sparse_categorical_accuracy: 1.0000\n", + "Epoch 9: val_sparse_categorical_accuracy improved from 0.86900 to 0.87800, saving model to training_1/checkpoint.model.keras\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.0457 - sparse_categorical_accuracy: 0.9974 - val_loss: 0.4145 - val_sparse_categorical_accuracy: 0.8780\n", + "Epoch 10/10\n", + "\u001b[1m 1/32\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 13ms/step - loss: 0.0286 - sparse_categorical_accuracy: 1.0000\n", + "Epoch 10: val_sparse_categorical_accuracy did not improve from 0.87800\n", + "\u001b[1m32/32\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 1ms/step - loss: 0.0351 - sparse_categorical_accuracy: 0.9987 - val_loss: 0.4200 - val_sparse_categorical_accuracy: 0.8720\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "checkpoint_path = \"training_1/checkpoint.model.keras\"\n", + "checkpoint_dir = os.path.dirname(checkpoint_path)\n", + "\n", + "# Create a callback that saves the model's weights\n", + "cp_callback = tf.keras.callbacks.ModelCheckpoint(filepath=checkpoint_path,\n", + " monitor='val_sparse_categorical_accuracy',\n", + " mode='max',\n", + " save_best_only=True,\n", + " verbose=1)\n", + "\n", + "# Train the model with the new callback\n", + "model.fit(train_images, \n", + " train_labels, \n", + " epochs=10,\n", + " validation_data=(test_images, test_labels),\n", + " callbacks=[cp_callback]) # Pass callback to training\n", + "\n", + "# This may generate warnings related to saving the state of the optimizer.\n", + "# These warnings (and similar warnings throughout this notebook)\n", + "# are in place to discourage outdated usage, and can be ignored." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "310eae08", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['checkpoint.model.keras']" + ] + }, + "execution_count": 8, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "os.listdir(checkpoint_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "50eeb6e5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "INFO:tensorflow:Assets written to: mnist_model/assets\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "INFO:tensorflow:Assets written to: mnist_model/assets\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Saved artifact at 'mnist_model'. The following endpoints are available:\n", + "\n", + "* Endpoint 'serve'\n", + " args_0 (POSITIONAL_ONLY): TensorSpec(shape=(None, 784), dtype=tf.float32, name='keras_tensor')\n", + "Output Type:\n", + " TensorSpec(shape=(None, 10), dtype=tf.float32, name=None)\n", + "Captures:\n", + " 139403584120848: TensorSpec(shape=(), dtype=tf.resource, name=None)\n", + " 139403240100240: TensorSpec(shape=(), dtype=tf.resource, name=None)\n", + " 139403240100048: TensorSpec(shape=(), dtype=tf.resource, name=None)\n", + " 139403240099856: TensorSpec(shape=(), dtype=tf.resource, name=None)\n" + ] + } + ], + "source": [ + "# Export model in saved_model format\n", + "model.export(\"mnist_model\")" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "6d3bba9e", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/rishic/anaconda3/envs/spark-dl-tf/lib/python3.11/site-packages/keras/src/layers/core/dense.py:87: UserWarning: Do not pass an `input_shape`/`input_dim` argument to a layer. When using Sequential models, prefer using an `Input(shape)` object as the first layer in the model instead.\n", + " super().__init__(activity_regularizer=activity_regularizer, **kwargs)\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "32/32 - 0s - 10ms/step - loss: 2.4196 - sparse_categorical_accuracy: 0.0590\n", + "Untrained model, accuracy: 5.90%\n" + ] + } + ], + "source": [ + "# Create a basic model instance\n", + "model = create_model()\n", + "\n", + "# Evaluate the model\n", + "loss, acc = model.evaluate(test_images, test_labels, verbose=2)\n", + "print(\"Untrained model, accuracy: {:5.2f}%\".format(100 * acc))" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "22ad1708", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "32/32 - 0s - 713us/step - loss: 0.4145 - sparse_categorical_accuracy: 0.8780\n", + "Restored model, accuracy: 87.80%\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "/home/rishic/anaconda3/envs/spark-dl-tf/lib/python3.11/site-packages/keras/src/saving/saving_lib.py:713: UserWarning: Skipping variable loading for optimizer 'adam', because it has 2 variables whereas the saved optimizer has 10 variables. \n", + " saveable.load_own_variables(weights_store.get(inner_path))\n" + ] + } + ], + "source": [ + "# Load the weights from the checkpoint\n", + "model.load_weights(checkpoint_path)\n", + "\n", + "# Re-evaluate the model\n", + "loss, acc = model.evaluate(test_images, test_labels, verbose=2)\n", + "print(\"Restored model, accuracy: {:5.2f}%\".format(100 * acc))" + ] + }, + { + "cell_type": "markdown", + "id": "1c097d63", + "metadata": {}, + "source": [ + "### Checkpoint callback options" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "cb336e89", + "metadata": {}, + "outputs": [], + "source": [ + "!rm -rf training_2\n", + "!mkdir training_2" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "750b6deb", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\n", + "Epoch 5: saving model to training_2/cp-0005.weights.h5\n", + "\n", + "Epoch 10: saving model to training_2/cp-0010.weights.h5\n", + "\n", + "Epoch 15: saving model to training_2/cp-0015.weights.h5\n", + "\n", + "Epoch 20: saving model to training_2/cp-0020.weights.h5\n", + "\n", + "Epoch 25: saving model to training_2/cp-0025.weights.h5\n", + "\n", + "Epoch 30: saving model to training_2/cp-0030.weights.h5\n", + "\n", + "Epoch 35: saving model to training_2/cp-0035.weights.h5\n", + "\n", + "Epoch 40: saving model to training_2/cp-0040.weights.h5\n", + "\n", + "Epoch 45: saving model to training_2/cp-0045.weights.h5\n", + "\n", + "Epoch 50: saving model to training_2/cp-0050.weights.h5\n" + ] + }, + { + "data": { + "text/plain": [ + "" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# Include the epoch in the file name (uses `str.format`)\n", + "checkpoint_path = \"training_2/cp-{epoch:04d}.weights.h5\"\n", + "checkpoint_dir = os.path.dirname(checkpoint_path)\n", + "\n", + "batch_size = 32\n", + "\n", + "# Calculate the number of batches per epoch\n", + "import math\n", + "n_batches = len(train_images) / batch_size\n", + "n_batches = math.ceil(n_batches) # round up the number of batches to the nearest whole integer\n", + "\n", + "# Create a callback that saves the model's weights every 5 epochs\n", + "cp_callback = tf.keras.callbacks.ModelCheckpoint(\n", + " filepath=checkpoint_path, \n", + " verbose=1, \n", + " save_weights_only=True,\n", + " save_freq=5*n_batches)\n", + "\n", + "# Create a new model instance\n", + "model = create_model()\n", + "\n", + "# Save the weights using the `checkpoint_path` format\n", + "model.save_weights(checkpoint_path.format(epoch=0))\n", + "\n", + "# Train the model with the new callback\n", + "model.fit(train_images, \n", + " train_labels,\n", + " epochs=50, \n", + " batch_size=batch_size, \n", + " callbacks=[cp_callback],\n", + " validation_data=(test_images, test_labels),\n", + " verbose=0)" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "1c43fd3d", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['cp-0000.weights.h5',\n", + " 'cp-0015.weights.h5',\n", + " 'cp-0010.weights.h5',\n", + " 'cp-0035.weights.h5',\n", + " 'cp-0020.weights.h5',\n", + " 'cp-0040.weights.h5',\n", + " 'cp-0050.weights.h5',\n", + " 'cp-0005.weights.h5',\n", + " 'cp-0045.weights.h5',\n", + " 'cp-0025.weights.h5',\n", + " 'cp-0030.weights.h5']" + ] + }, + "execution_count": 14, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "os.listdir(checkpoint_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "0d7ae715", + "metadata": {}, + "outputs": [], + "source": [ + "latest = \"training_2/cp-0030.weights.h5\"" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "d345c6f7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "32/32 - 0s - 9ms/step - loss: 0.4501 - sparse_categorical_accuracy: 0.8720\n", + "Restored model, accuracy: 87.20%\n" + ] + } + ], + "source": [ + "# Create a new model instance\n", + "model = create_model()\n", + "\n", + "# Load the previously saved weights\n", + "model.load_weights(latest)\n", + "\n", + "# Re-evaluate the model from the latest checkpoint\n", + "loss, acc = model.evaluate(test_images, test_labels, verbose=2)\n", + "print(\"Restored model, accuracy: {:5.2f}%\".format(100 * acc))" + ] + }, + { + "cell_type": "markdown", + "id": "a86f4700", + "metadata": {}, + "source": [ + "## PySpark" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "7fcf07bb", + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2c022c24", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" + ] + }, + { + "cell_type": "markdown", + "id": "c81d0b1b", + "metadata": {}, + "source": [ + "### Convert numpy array to Spark DataFrame (via Pandas DataFrame)" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "49ff5203", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "(1000, 784)" + ] + }, + "execution_count": 19, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# numpy array to pandas DataFrame\n", + "test_pdf = pd.DataFrame(test_images)\n", + "test_pdf.shape" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "182ee0c7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 134 ms, sys: 15.5 ms, total: 149 ms\n", + "Wall time: 1.36 s\n" + ] + } + ], + "source": [ + "%%time\n", + "df = spark.createDataFrame(test_pdf).repartition(8)" + ] + }, + { + "cell_type": "markdown", + "id": "d4e1c7ec-64fa-43c4-9bcf-0868a401d1f2", + "metadata": {}, + "source": [ + "### Save as Parquet (784 columns of float)" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "0061c39a-0871-429e-a4ff-751d26bf4b04", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "24/10/03 17:40:32 WARN SparkStringUtils: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.\n", + "[Stage 0:> (0 + 8) / 8]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 2.49 ms, sys: 1.65 ms, total: 4.13 ms\n", + "Wall time: 1.66 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "df.write.mode(\"overwrite\").parquet(\"mnist_784\")" + ] + }, + { + "cell_type": "markdown", + "id": "18315afb-3fa2-4953-9297-52c04dd70c32", + "metadata": {}, + "source": [ + "### Save as Parquet (1 column of 784 float)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "302c73ec", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 6.71 ms, sys: 4.92 ms, total: 11.6 ms\n", + "Wall time: 11.4 ms\n" + ] + }, + { + "data": { + "text/plain": [ + "(1000, 1)" + ] + }, + "execution_count": 22, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "%%time\n", + "test_pdf['data'] = test_pdf.values.tolist()\n", + "pdf = test_pdf[['data']]\n", + "pdf.shape" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "5495901b", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 46.6 ms, sys: 4.71 ms, total: 51.3 ms\n", + "Wall time: 91.7 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "df = spark.createDataFrame(pdf)" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "5fa7faa8-c6bd-41b0-b5f7-fb121f0332e6", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 807 μs, sys: 724 μs, total: 1.53 ms\n", + "Wall time: 211 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "df.write.mode(\"overwrite\").parquet(\"mnist_1\")" + ] + }, + { + "cell_type": "markdown", + "id": "c87b444e", + "metadata": {}, + "source": [ + "### Check arrow memory configuration" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "3d4ca414", + "metadata": {}, + "outputs": [], + "source": [ + "spark.conf.set(\"spark.sql.execution.arrow.maxRecordsPerBatch\", \"128\")\n", + "# This line will fail if the vectorized reader runs out of memory\n", + "assert len(df.head()) > 0, \"`df` should not be empty\" " + ] + }, + { + "cell_type": "markdown", + "id": "9b6dde30-98a9-45db-ab3a-d4546f9bed99", + "metadata": {}, + "source": [ + "## Inference using Spark DL API" + ] + }, + { + "cell_type": "markdown", + "id": "4238fb28-d002-4b4d-9aa1-8af1fbd5d569", + "metadata": {}, + "source": [ + "### 1 column of 784 float" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "db30fba6-24d0-4c00-8502-04f9b10e7e16", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import os\n", + "import pandas as pd\n", + "\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import array, col, struct\n", + "from pyspark.sql.types import ArrayType, FloatType, Union, Dict" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "b9cf62f8-96b2-4716-80bd-bb93d5f939bd", + "metadata": {}, + "outputs": [], + "source": [ + "# get absolute path to model\n", + "model_dir = \"{}/training_1/checkpoint.model.keras\".format(os.getcwd())" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "b81fa297-d9d0-4600-880d-dbdcdf8bccc6", + "metadata": {}, + "outputs": [], + "source": [ + "def predict_batch_fn():\n", + " import tensorflow as tf\n", + "\n", + " # Enable GPU memory growth to avoid CUDA OOM\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + "\n", + " model = tf.keras.models.load_model(model_dir)\n", + " def predict(inputs: np.ndarray) -> np.ndarray:\n", + " return model.predict(inputs)\n", + " \n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "72a689bd-dd82-492e-8740-1738a215325f", + "metadata": {}, + "outputs": [], + "source": [ + "mnist = predict_batch_udf(predict_batch_fn,\n", + " return_type=ArrayType(FloatType()),\n", + " batch_size=1024,\n", + " input_tensor_shapes=[[784]])" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "60a70150-26b1-4145-9e7d-6e17389216b7", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "1" + ] + }, + "execution_count": 30, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = spark.read.parquet(\"mnist_1\")\n", + "len(df.columns)" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "e027f0d2-0f65-47b7-a562-2f0965faceec", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+--------------------+\n", + "| data|\n", + "+--------------------+\n", + "|[0.0, 0.0, 0.0, 0...|\n", + "|[0.0, 0.0, 0.0, 0...|\n", + "|[0.0, 0.0, 0.0, 0...|\n", + "|[0.0, 0.0, 0.0, 0...|\n", + "|[0.0, 0.0, 0.0, 0...|\n", + "+--------------------+\n", + "only showing top 5 rows\n", + "\n" + ] + } + ], + "source": [ + "df.show(5)" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "f0c3fb2e-469e-47bc-b948-8f6b0d7f6513", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 4:===================================================> (7 + 1) / 8]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 18.5 ms, sys: 13.3 ms, total: 31.8 ms\n", + "Wall time: 5.03 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "# first pass caches model/fn\n", + "preds = df.withColumn(\"preds\", mnist(struct(df.columns))).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "cdfa229a-f4a9-4c11-a410-de4a21c02c82", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 37.3 ms, sys: 12.4 ms, total: 49.8 ms\n", + "Wall time: 259 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(*df.columns)).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "5586ce49-6f93-4343-9b66-0dbb64972179", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 22.9 ms, sys: 5.96 ms, total: 28.8 ms\n", + "Wall time: 237 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(*[col(c) for c in df.columns])).collect()" + ] + }, + { + "cell_type": "markdown", + "id": "004f1599-3c62-499e-9fd8-ed5cb0c90de4", + "metadata": { + "tags": [] + }, + "source": [ + "#### Check predictions" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "4f947dc0-6b18-4605-810b-e83250a161db", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
datapreds
0[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-5.88436, -3.1058547, 0.10873719, 12.67319, -...
1[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-3.273286, -8.362554, 1.8936121, -3.8881433, ...
2[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-3.3856308, 0.6785604, 1.3146863, 0.9275978, ...
3[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-2.7754688, -7.3659225, 11.768427, 1.3434286,...
4[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-4.9426627, 4.0774136, -0.4529277, -0.9312789...
5[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-5.226616, -3.1389174, 2.6100307, 3.695045, -...
6[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-4.3006196, 5.1169925, 0.5850615, -0.76248693...
7[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-2.3985956, -1.4814724, -4.884057, -0.2391600...
8[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[0.82160115, -2.8640625, -1.6951559, -4.489290...
9[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-1.2338604, -2.151981, -4.171742, 1.6106845, ...
\n", + "
" + ], + "text/plain": [ + " data \\\n", + "0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "1 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "2 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "3 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "4 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "5 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "6 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "7 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "8 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "9 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "\n", + " preds \n", + "0 [-5.88436, -3.1058547, 0.10873719, 12.67319, -... \n", + "1 [-3.273286, -8.362554, 1.8936121, -3.8881433, ... \n", + "2 [-3.3856308, 0.6785604, 1.3146863, 0.9275978, ... \n", + "3 [-2.7754688, -7.3659225, 11.768427, 1.3434286,... \n", + "4 [-4.9426627, 4.0774136, -0.4529277, -0.9312789... \n", + "5 [-5.226616, -3.1389174, 2.6100307, 3.695045, -... \n", + "6 [-4.3006196, 5.1169925, 0.5850615, -0.76248693... \n", + "7 [-2.3985956, -1.4814724, -4.884057, -0.2391600... \n", + "8 [0.82160115, -2.8640625, -1.6951559, -4.489290... \n", + "9 [-1.2338604, -2.151981, -4.171742, 1.6106845, ... " + ] + }, + "execution_count": 35, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "preds = df.withColumn(\"preds\", mnist(*df.columns)).limit(10).toPandas()\n", + "preds" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "de4964e0-d1f8-4753-afa1-a8f95ca3f151", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([ -5.88436 , -3.1058547 , 0.10873719, 12.67319 ,\n", + " -5.143787 , 4.0859914 , -10.203137 , -1.4333997 ,\n", + " -3.3865087 , -3.8473575 ], dtype=float32)" + ] + }, + "execution_count": 36, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sample = preds.iloc[0]\n", + "sample.preds" + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "44e9a874-e301-4b72-8df7-bf1c5133c287", + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np" + ] + }, + { + "cell_type": "code", + "execution_count": 38, + "id": "c60e5af4-fc1e-4575-a717-f304664235be", + "metadata": {}, + "outputs": [], + "source": [ + "prediction = np.argmax(sample.preds)\n", + "img = np.array(sample.data).reshape(28,28)" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "eb45ecc9-d376-40c4-ad7b-2bd08ca5aaf6", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAkfElEQVR4nO3dfXQUdZ7v8U/nqSEkaR7yLAFCFHRAcAYly/AgSiQEZUCYGUG9F7gziJiggI6KR0Udzsksrg7qIHjcHVhHEGWOyMoiDg9JGBRwwTCIM2QhJ0g4kIBcSYcAIaR/9w+uvbQkQDUdfkl4v86pc+iq37fqm6Lgk+qqrnYZY4wAALjKwmw3AAC4NhFAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAgAPdunXTpEmT/K8LCwvlcrlUWFgYsm24XC698MILIVsf0FwRQGgxlixZIpfL5Z/atGmjHj16KC8vT5WVlbbbc2TNmjUtJmTefvtt3X777UpKSpLb7VZ6eromT56s/fv3224NLVyE7QYAp1566SWlp6fr9OnT2rx5sxYuXKg1a9Zo9+7dio6Ovqq9DBkyRKdOnVJUVJSjujVr1mjBggUNhtCpU6cUEdF8/mkWFxcrPT1dP/vZz9ShQweVlZXp7bff1urVq/W3v/1NqamptltEC9V8jnLgMuXk5OjWW2+VJP36179Wp06d9Oqrr2rVqlWaMGFCgzU1NTVq165dyHsJCwtTmzZtQrrOUK/vSr355psXzBszZoxuvfVWvfPOO3r66actdIXWgLfg0OLdeeedkqSysjJJ0qRJkxQTE6PS0lKNHDlSsbGxeuCBByRJPp9P8+fPV69evdSmTRslJSVp6tSp+u677wLWaYzR3Llz1blzZ0VHR+uOO+7Q119/fcG2G7sGtG3bNo0cOVIdOnRQu3bt1KdPH7322mv+/hYsWCBJAW8pfq+ha0DFxcXKyclRXFycYmJiNGzYMG3dujVgzPdvUX722WeaNWuWEhIS1K5dO9177706evRowNiqqirt2bNHVVVVl7OLL9CtWzdJ0vHjx4OqByTOgNAKlJaWSpI6derkn3f27FllZ2dr0KBB+pd/+Rf/W3NTp07VkiVLNHnyZD366KMqKyvTH/7wBxUXF+uzzz5TZGSkJOn555/X3LlzNXLkSI0cOVJffvmlhg8frjNnzlyyn3Xr1umee+5RSkqKHnvsMSUnJ+sf//iHVq9erccee0xTp07VoUOHtG7dOv3pT3+65Pq+/vprDR48WHFxcXryyScVGRmpt956S0OHDlVRUZEyMzMDxk+fPl0dOnTQnDlztH//fs2fP195eXl6//33/WNWrlypyZMna/HixQE3VVzMsWPHVF9frwMHDuill16SJA0bNuyyaoEGGaCFWLx4sZFk1q9fb44ePWrKy8vN8uXLTadOnUzbtm3NwYMHjTHGTJw40UgyTz/9dED9X//6VyPJLF26NGD+2rVrA+YfOXLEREVFmbvvvtv4fD7/uGeeecZIMhMnTvTPKygoMJJMQUGBMcaYs2fPmvT0dNO1a1fz3XffBWzn/HXl5uaaxv75STJz5szxvx4zZoyJiooypaWl/nmHDh0ysbGxZsiQIRfsn6ysrIBtzZw504SHh5vjx49fMHbx4sUN9tAQt9ttJBlJplOnTub111+/7FqgIbwFhxYnKytLCQkJSktL0/jx4xUTE6OVK1fquuuuCxg3bdq0gNcrVqyQx+PRXXfdpW+//dY/9evXTzExMSooKJAkrV+/XmfOnNH06dMD3hqbMWPGJXsrLi5WWVmZZsyYofbt2wcsO39dl6u+vl5/+ctfNGbMGHXv3t0/PyUlRffff782b94sr9cbUPPQQw8FbGvw4MGqr6/XN9984583adIkGWMu++xHkj755BOtWbNGr7zyirp06aKamhrHPw9wPt6CQ4uzYMEC9ejRQxEREUpKSlLPnj0VFhb4u1RERIQ6d+4cMG/v3r2qqqpSYmJig+s9cuSIJPn/o77hhhsClickJKhDhw4X7e37twN79+59+T/QRRw9elQnT55Uz549L1h20003yefzqby8XL169fLP79KlS8C473v+4XUup+644w5J524CGT16tHr37q2YmBjl5eVd0Xpx7SKA0OL079/ffxdcY9xu9wWh5PP5lJiYqKVLlzZYk5CQELIebQoPD29wvjEmZNvIyMjQj3/8Yy1dupQAQtAIIFwzMjIytH79eg0cOFBt27ZtdFzXrl0lnTtjOv9tr6NHj17yLCIjI0OStHv3bmVlZTU67nLfjktISFB0dLRKSkouWLZnzx6FhYUpLS3tstYVaqdOnVJtba2VbaN14BoQrhm//OUvVV9fr9/+9rcXLDt79qz/luKsrCxFRkbqjTfeCDhrmD9//iW38ZOf/ETp6emaP3/+Bbcon7+u7z+TdKnbmMPDwzV8+HCtWrUq4MkDlZWVWrZsmQYNGqS4uLhL9vVDl3sb9tmzZxsM3S+++EJfffXVJc9EgYvhDAjXjNtvv11Tp05Vfn6+du7cqeHDhysyMlJ79+7VihUr9Nprr+nnP/+5EhIS9MQTTyg/P1/33HOPRo4cqeLiYn3yySeKj4+/6DbCwsK0cOFCjRo1SrfccosmT56slJQU7dmzR19//bU+/fRTSVK/fv0kSY8++qiys7MVHh6u8ePHN7jOuXPnat26dRo0aJAeeeQRRURE6K233lJtba3mzZsX1L643NuwT5w4obS0NN13333q1auX2rVrp6+++kqLFy+Wx+PRc889F9T2AYkAwjVm0aJF6tevn9566y0988wzioiIULdu3fTggw9q4MCB/nFz585VmzZttGjRIhUUFCgzM1N/+ctfdPfdd19yG9nZ2SooKNCLL76oV155RT6fTxkZGZoyZYp/zNixYzV9+nQtX75c7777rowxjQZQr1699Ne//lWzZ89Wfn6+fD6fMjMz9e67717wGaBQi46O1q9//WsVFBToz3/+s06dOqXU1FRNmDBBzz77rP8DqUAwXCaUVyYBALhMXAMCAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCKZvc5IJ/Pp0OHDik2NjaopwcDAOwyxqi6ulqpqakXPJPxfM0ugA4dOmTt2VYAgNApLy+/4Kn052t2ARQbGytJGqSRilCk5W4AAE6dVZ02a43///PGNFkALViwQC+//LIqKirUt29fvfHGG+rfv/8l675/2y1CkYpwEUAA0OL8/+frXOoySpPchPD+++9r1qxZmjNnjr788kv17dtX2dnZ/i/8AgCgSQLo1Vdf1ZQpUzR58mT96Ec/0qJFixQdHa0//vGPTbE5AEALFPIAOnPmjHbs2BHwZVxhYWHKysrSli1bLhhfW1srr9cbMAEAWr+QB9C3336r+vp6JSUlBcxPSkpSRUXFBePz8/Pl8Xj8E3fAAcC1wfoHUWfPnq2qqir/VF5ebrslAMBVEPK74OLj4xUeHq7KysqA+ZWVlUpOTr5gvNvtltvtDnUbAIBmLuRnQFFRUerXr582bNjgn+fz+bRhwwYNGDAg1JsDALRQTfI5oFmzZmnixIm69dZb1b9/f82fP181NTWaPHlyU2wOANACNUkA3XfffTp69Kief/55VVRU6JZbbtHatWsvuDEBAHDtchljjO0mzuf1euXxeDRUo3kSAgC0QGdNnQq1SlVVVYqLi2t0nPW74AAA1yYCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWBFhuwHYZ37aN7i6cOe/v0RWeh3XlP7vRMc1vu6nHNdI0p7b/+i4JtzlfD9MPTjAcU3hp7c4run2nzWOayRJW3cFVwc4wBkQAMAKAggAYEXIA+iFF16Qy+UKmG688cZQbwYA0MI1yTWgXr16af369f+zkQguNQEAAjVJMkRERCg5ObkpVg0AaCWa5BrQ3r17lZqaqu7du+uBBx7QgQMHGh1bW1srr9cbMAEAWr+QB1BmZqaWLFmitWvXauHChSorK9PgwYNVXV3d4Pj8/Hx5PB7/lJaWFuqWAADNUMgDKCcnR7/4xS/Up08fZWdna82aNTp+/Lg++OCDBsfPnj1bVVVV/qm8vDzULQEAmqEmvzugffv26tGjh/bt29fgcrfbLbfb3dRtAACamSb/HNCJEydUWlqqlJSUpt4UAKAFCXkAPfHEEyoqKtL+/fv1+eef695771V4eLgmTJgQ6k0BAFqwkL8Fd/DgQU2YMEHHjh1TQkKCBg0apK1btyohISHUmwIAtGAuY4yx3cT5vF6vPB6Phmq0IlyRttuxqubnmY5rKm91flK7dsLLjmskqUtEW8c1/2v/XY5r/tRtneManFN8xhdU3eOP5zmuif5wW1DbQutz1tSpUKtUVVWluLi4RsfxLDgAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIKHkV4lR/J+6rim8OlXHNdEu6Ic1zR339afclzTxhXc71Z1cv7PYcaBexzX/DLxvxzX3B1d5bgmWPvqah3XPDH4l45rzpYfdFyD5o+HkQIAmjUCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsiLDdwLXCF+68pjU+2frlYz9yXLNhxiDHNfVtg/vd6rvrnT+B/br/POy45s2EcY5r7v7zHx3XBGvsf011XNPt+P7QN4JWjTMgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCh5FeJan/+jfHNR88kui4Jjv6gOOanDlPOK6RpLoYl+Oa6/7joOOaiP07nNc4rjgnOYia+iBqKu/5aRBVV8+uny5xXDMmiAes+qqrHdeg9eAMCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCs4GGkV4mvpsZxzTs90xzXvJ0z1nFNfEGx4xpJ8p0+7bjmbFBbunrCExIc13x3V4bjmscf/sBxDdDacAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcByY4yef/55paSkqG3btsrKytLevXtD1S8AoJVwHEA1NTXq27evFixY0ODyefPm6fXXX9eiRYu0bds2tWvXTtnZ2TodxPUCAEDr5fgmhJycHOXk5DS4zBij+fPn69lnn9Xo0aMlSe+8846SkpL00Ucfafz48VfWLQCg1QjpNaCysjJVVFQoKyvLP8/j8SgzM1NbtmxpsKa2tlZerzdgAgC0fiENoIqKCklSUlJSwPykpCT/sh/Kz8+Xx+PxT2lpzm89BgC0PNbvgps9e7aqqqr8U3l5ue2WAABXQUgDKDk5WZJUWVkZML+ystK/7Ifcbrfi4uICJgBA6xfSAEpPT1dycrI2bNjgn+f1erVt2zYNGDAglJsCALRwju+CO3HihPbt2+d/XVZWpp07d6pjx47q0qWLZsyYoblz5+qGG25Qenq6nnvuOaWmpmrMmDGh7BsA0MI5DqDt27frjjvu8L+eNWuWJGnixIlasmSJnnzySdXU1Oihhx7S8ePHNWjQIK1du1Zt2rQJXdcAgBbPZYwxtps4n9frlcfj0VCNVoQr0nY7aKHC23uCqnt8x2bHNUPanAlqW1eDT76g6n73bV/HNduGpTiuqf/2mOMaNH9nTZ0KtUpVVVUXva5v/S44AMC1iQACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACscfx0D0BKUPdorqLohbTaGuBO7VtXEB1X3ed+oIKp4sjWc4QwIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKzgYaRAK3ZX28NB1c2d9YDjmrqYoDblWGLxWcc1bT7+ogk6wZXiDAgAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArOBhpGiV0jacDKpuxyTnNf3cQW3qqogJC665HY+/EeJOQmfOkR87rtnxMb9rN0f8rQAArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFTyMFK2S67OdQdW9NOp+xzW1STGOa6of9zqu+eyW5Y5rWqNnE7Y7rrnzwUeD2pbn3a1B1eHycAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcDySZMmyeVyBUwjRowIVb8AgFbCcQDV1NSob9++WrBgQaNjRowYocOHD/un995774qaBAC0Po5vQsjJyVFOTs5Fx7jdbiUnJwfdFACg9WuSa0CFhYVKTExUz549NW3aNB07dqzRsbW1tfJ6vQETAKD1C3kAjRgxQu+88442bNigf/7nf1ZRUZFycnJUX1/f4Pj8/Hx5PB7/lJaWFuqWAADNUMg/BzR+/Hj/n2+++Wb16dNHGRkZKiws1LBhwy4YP3v2bM2aNcv/2uv1EkIAcA1o8tuwu3fvrvj4eO3bt6/B5W63W3FxcQETAKD1a/IAOnjwoI4dO6aUlJSm3hQAoAVx/BbciRMnAs5mysrKtHPnTnXs2FEdO3bUiy++qHHjxik5OVmlpaV68skndf311ys7OzukjQMAWjbHAbR9+3bdcccd/tffX7+ZOHGiFi5cqF27dunf//3fdfz4caWmpmr48OH67W9/K7fbHbquAQAtnuMAGjp0qIwxjS7/9NNPr6ghwKb6r0sc10R87Xw7HQpcjmtGRf3Ucc3+P/VwXCNJn2QudFzTOaJtUNtyKtIV7rjmdMfgrjZ4gqrC5eJZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAi5F/JDeAyXOSJ8o2W1NY6run6y68c10jSnW/NdFzz3/csCmpbuHZxBgQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVvAwUqAVc0VGBVfXtj7EnYTOrjPOe0vcXtMEneBKcQYEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFbwMFKgFSt545ag6v572MLQNhJCMx6f7rgm+vNtTdAJrhRnQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQ8jRasUFhsbXF17T4g7adiRu9Ic19w1/TPHNf+RuMBxzTlX53fTD04kOq6J2/KN45qzjitwNXAGBACwggACAFjhKIDy8/N12223KTY2VomJiRozZoxKSkoCxpw+fVq5ubnq1KmTYmJiNG7cOFVWVoa0aQBAy+cogIqKipSbm6utW7dq3bp1qqur0/Dhw1VTU+MfM3PmTH388cdasWKFioqKdOjQIY0dOzbkjQMAWjZHNyGsXbs24PWSJUuUmJioHTt2aMiQIaqqqtK//du/admyZbrzzjslSYsXL9ZNN92krVu36p/+6Z9C1zkAoEW7omtAVVVVkqSOHTtKknbs2KG6ujplZWX5x9x4443q0qWLtmzZ0uA6amtr5fV6AyYAQOsXdAD5fD7NmDFDAwcOVO/evSVJFRUVioqKUvv27QPGJiUlqaKiosH15Ofny+Px+Ke0NOe3pwIAWp6gAyg3N1e7d+/W8uXLr6iB2bNnq6qqyj+Vl5df0foAAC1DUB9EzcvL0+rVq7Vp0yZ17tzZPz85OVlnzpzR8ePHA86CKisrlZyc3OC63G633G53MG0AAFowR2dAxhjl5eVp5cqV2rhxo9LT0wOW9+vXT5GRkdqwYYN/XklJiQ4cOKABAwaEpmMAQKvg6AwoNzdXy5Yt06pVqxQbG+u/ruPxeNS2bVt5PB796le/0qxZs9SxY0fFxcVp+vTpGjBgAHfAAQACOAqghQsXSpKGDh0aMH/x4sWaNGmSJOn3v/+9wsLCNG7cONXW1io7O1tvvvlmSJoFALQeLmOMsd3E+bxerzwej4ZqtCJckbbbuSaE9b0pqLo9uTGOa5LT/q/jmiMlCY5rJt9Z6LhGkp7q9HVQdQhOn88nOa7p8ouvQt8IQuqsqVOhVqmqqkpxcXGNjuNZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAiqG9ERfPl6tfLcU3b3x8Jalv/nfFuUHWO9bk6m2nuak2d45pIV3hQ26qsr3VcM+dQjuOazq8F1x9aB86AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKHkbaytR0jXFc8373fw1ya1FB1jU9n3xB1c08NNhxzW8S1zuuyf4813FNbGG045rqbo5LJEnps7cEUVXtuCJMO4PYDloLzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoeRtrKRH+4zXHNLw4/HNS2jv64neMaXxDPL61z/nxVvf1//uC8SFLpbacd10z78VTHNek7dzmukTGOS+KdbwW4ajgDAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArXMYE8YTDJuT1euXxeDRUoxXhirTdDgDAobOmToVapaqqKsXFxTU6jjMgAIAVBBAAwApHAZSfn6/bbrtNsbGxSkxM1JgxY1RSUhIwZujQoXK5XAHTww8H930zAIDWy1EAFRUVKTc3V1u3btW6detUV1en4cOHq6amJmDclClTdPjwYf80b968kDYNAGj5HH0j6tq1awNeL1myRImJidqxY4eGDBninx8dHa3k5OTQdAgAaJWu6BpQVVWVJKljx44B85cuXar4+Hj17t1bs2fP1smTJxtdR21trbxeb8AEAGj9HJ0Bnc/n82nGjBkaOHCgevfu7Z9///33q2vXrkpNTdWuXbv01FNPqaSkRB9++GGD68nPz9eLL74YbBsAgBYq6M8BTZs2TZ988ok2b96szp07Nzpu48aNGjZsmPbt26eMjIwLltfW1qq2ttb/2uv1Ki0tjc8BAUALdbmfAwrqDCgvL0+rV6/Wpk2bLho+kpSZmSlJjQaQ2+2W2+0Opg0AQAvmKICMMZo+fbpWrlypwsJCpaenX7Jm586dkqSUlJSgGgQAtE6OAig3N1fLli3TqlWrFBsbq4qKCkmSx+NR27ZtVVpaqmXLlmnkyJHq1KmTdu3apZkzZ2rIkCHq06dPk/wAAICWydE1IJfL1eD8xYsXa9KkSSovL9eDDz6o3bt3q6amRmlpabr33nv17LPPXvR9wPPxLDgAaNma5BrQpbIqLS1NRUVFTlYJALhG8Sw4AIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVEbYb+CFjjCTprOokY7kZAIBjZ1Un6X/+P29Mswug6upqSdJmrbHcCQDgSlRXV8vj8TS63GUuFVFXmc/n06FDhxQbGyuXyxWwzOv1Ki0tTeXl5YqLi7PUoX3sh3PYD+ewH85hP5zTHPaDMUbV1dVKTU1VWFjjV3qa3RlQWFiYOnfufNExcXFx1/QB9j32wznsh3PYD+ewH86xvR8udubzPW5CAABYQQABAKxoUQHkdrs1Z84cud1u261YxX44h/1wDvvhHPbDOS1pPzS7mxAAANeGFnUGBABoPQggAIAVBBAAwAoCCABgBQEEALCixQTQggUL1K1bN7Vp00aZmZn64osvbLd01b3wwgtyuVwB04033mi7rSa3adMmjRo1SqmpqXK5XProo48Clhtj9PzzzyslJUVt27ZVVlaW9u7da6fZJnSp/TBp0qQLjo8RI0bYabaJ5Ofn67bbblNsbKwSExM1ZswYlZSUBIw5ffq0cnNz1alTJ8XExGjcuHGqrKy01HHTuJz9MHTo0AuOh4cffthSxw1rEQH0/vvva9asWZozZ46+/PJL9e3bV9nZ2Tpy5Ijt1q66Xr166fDhw/5p8+bNtltqcjU1Nerbt68WLFjQ4PJ58+bp9ddf16JFi7Rt2za1a9dO2dnZOn369FXutGldaj9I0ogRIwKOj/fee+8qdtj0ioqKlJubq61bt2rdunWqq6vT8OHDVVNT4x8zc+ZMffzxx1qxYoWKiop06NAhjR071mLXoXc5+0GSpkyZEnA8zJs3z1LHjTAtQP/+/U1ubq7/dX19vUlNTTX5+fkWu7r65syZY/r27Wu7DaskmZUrV/pf+3w+k5ycbF5++WX/vOPHjxu3223ee+89Cx1eHT/cD8YYM3HiRDN69Ggr/dhy5MgRI8kUFRUZY8793UdGRpoVK1b4x/zjH/8wksyWLVtstdnkfrgfjDHm9ttvN4899pi9pi5Dsz8DOnPmjHbs2KGsrCz/vLCwMGVlZWnLli0WO7Nj7969Sk1NVffu3fXAAw/owIEDtluyqqysTBUVFQHHh8fjUWZm5jV5fBQWFioxMVE9e/bUtGnTdOzYMdstNamqqipJUseOHSVJO3bsUF1dXcDxcOONN6pLly6t+nj44X743tKlSxUfH6/evXtr9uzZOnnypI32GtXsnob9Q99++63q6+uVlJQUMD8pKUl79uyx1JUdmZmZWrJkiXr27KnDhw/rxRdf1ODBg7V7927Fxsbabs+KiooKSWrw+Ph+2bVixIgRGjt2rNLT01VaWqpnnnlGOTk52rJli8LDw223F3I+n08zZszQwIED1bt3b0nnjoeoqCi1b98+YGxrPh4a2g+SdP/996tr165KTU3Vrl279NRTT6mkpEQffvihxW4DNfsAwv/Iycnx/7lPnz7KzMxU165d9cEHH+hXv/qVxc7QHIwfP97/55tvvll9+vRRRkaGCgsLNWzYMIudNY3c3Fzt3r37mrgOejGN7YeHHnrI/+ebb75ZKSkpGjZsmEpLS5WRkXG122xQs38LLj4+XuHh4RfcxVJZWank5GRLXTUP7du3V48ePbRv3z7brVjz/THA8XGh7t27Kz4+vlUeH3l5eVq9erUKCgoCvj8sOTlZZ86c0fHjxwPGt9bjobH90JDMzExJalbHQ7MPoKioKPXr108bNmzwz/P5fNqwYYMGDBhgsTP7Tpw4odLSUqWkpNhuxZr09HQlJycHHB9er1fbtm275o+PgwcP6tixY63q+DDGKC8vTytXrtTGjRuVnp4esLxfv36KjIwMOB5KSkp04MCBVnU8XGo/NGTnzp2S1LyOB9t3QVyO5cuXG7fbbZYsWWL+/ve/m4ceesi0b9/eVFRU2G7tqnr88cdNYWGhKSsrM5999pnJysoy8fHx5siRI7Zba1LV1dWmuLjYFBcXG0nm1VdfNcXFxeabb74xxhjzu9/9zrRv396sWrXK7Nq1y4wePdqkp6ebU6dOWe48tC62H6qrq80TTzxhtmzZYsrKysz69evNT37yE3PDDTeY06dP2249ZKZNm2Y8Ho8pLCw0hw8f9k8nT570j3n44YdNly5dzMaNG8327dvNgAEDzIABAyx2HXqX2g/79u0zL730ktm+fbspKyszq1atMt27dzdDhgyx3HmgFhFAxhjzxhtvmC5dupioqCjTv39/s3XrVtstXXX33XefSUlJMVFRUea6664z9913n9m3b5/ttppcQUGBkXTBNHHiRGPMuVuxn3vuOZOUlGTcbrcZNmyYKSkpsdt0E7jYfjh58qQZPny4SUhIMJGRkaZr165mypQpre6XtIZ+fklm8eLF/jGnTp0yjzzyiOnQoYOJjo429957rzl8+LC9ppvApfbDgQMHzJAhQ0zHjh2N2+02119/vfnNb35jqqqq7Db+A3wfEADAimZ/DQgA0DoRQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAV/w/hgVLrpVGHsAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.figure()\n", + "plt.title(\"Prediction: {}\".format(prediction))\n", + "plt.imshow(img)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "39167347-0b99-4972-998c-e1230bf1d4d5", + "metadata": {}, + "source": [ + "### 784 columns of float" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "f1285e8b-1b96-437b-973a-eb868e33afb7", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import array, col, struct\n", + "from pyspark.sql.types import ArrayType, FloatType, Union, Dict" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "6bea332e-f6de-494f-a0db-795d9fe3e134", + "metadata": {}, + "outputs": [], + "source": [ + "def predict_batch_fn():\n", + " import tensorflow as tf\n", + " # Enable GPU memory growth\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + " \n", + " model = tf.keras.models.load_model(model_dir)\n", + " def predict(inputs: np.ndarray) -> np.ndarray:\n", + " return model.predict(inputs)\n", + " \n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "id": "731d234c-549f-4df3-8a2b-312e63195396", + "metadata": {}, + "outputs": [], + "source": [ + "mnist = predict_batch_udf(predict_batch_fn,\n", + " return_type=ArrayType(FloatType()),\n", + " batch_size=1024,\n", + " input_tensor_shapes=[[784]])" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "id": "a40fe207-6246-4b0e-abde-823979878d97", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "784" + ] + }, + "execution_count": 43, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = spark.read.parquet(\"mnist_784\")\n", + "len(df.columns)" + ] + }, + { + "cell_type": "code", + "execution_count": 44, + "id": "10904f12-03e7-4518-8f12-2aa11989ddf5", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 10:=======> (1 + 7) / 8]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 45.6 ms, sys: 26 ms, total: 71.6 ms\n", + "Wall time: 5.51 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(struct(*df.columns))).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "id": "671128df-f0f4-4f54-b35c-d63a78c7f89a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 46.5 ms, sys: 34 ms, total: 80.5 ms\n", + "Wall time: 884 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", mnist(array(*df.columns))).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "id": "ce35deaf-7d49-4f34-9bf9-b4e6fc5761f4", + "metadata": {}, + "outputs": [], + "source": [ + "# should raise ValueError\n", + "# preds = df.withColumn(\"preds\", mnist(*df.columns)).collect()" + ] + }, + { + "cell_type": "markdown", + "id": "01709833-484b-451f-9aa8-37be5b7baf14", + "metadata": {}, + "source": [ + "### Check prediction" + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "id": "f9119632-b284-45d7-a262-c262e034c15c", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
0123456789...775776777778779780781782783preds
00.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[-5.88436, -3.1058552, 0.108737305, 12.67319, ...
10.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[-3.2732859, -8.362555, 1.893612, -3.888143, 0...
20.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[-3.3856308, 0.6785604, 1.3146865, 0.9275978, ...
30.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[-2.775469, -7.3659234, 11.768431, 1.3434289, ...
40.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[-4.942663, 4.0774136, -0.45292768, -0.9312788...
50.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[-5.226616, -3.1389174, 2.6100307, 3.695045, -...
60.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[-4.3006196, 5.116993, 0.5850617, -0.7624871, ...
70.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[-2.398596, -1.4814726, -4.8840575, -0.2391601...
80.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[0.82160157, -2.8640628, -1.6951559, -4.489291...
90.00.00.00.00.00.00.00.00.00.0...0.00.00.00.00.00.00.00.00.0[-1.2338604, -2.151981, -4.1717424, 1.6106843,...
\n", + "

10 rows × 785 columns

\n", + "
" + ], + "text/plain": [ + " 0 1 2 3 4 5 6 7 8 9 ... 775 776 777 778 \\\n", + "0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "1 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "2 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "3 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "4 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "5 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "6 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "7 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "8 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "9 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 ... 0.0 0.0 0.0 0.0 \n", + "\n", + " 779 780 781 782 783 preds \n", + "0 0.0 0.0 0.0 0.0 0.0 [-5.88436, -3.1058552, 0.108737305, 12.67319, ... \n", + "1 0.0 0.0 0.0 0.0 0.0 [-3.2732859, -8.362555, 1.893612, -3.888143, 0... \n", + "2 0.0 0.0 0.0 0.0 0.0 [-3.3856308, 0.6785604, 1.3146865, 0.9275978, ... \n", + "3 0.0 0.0 0.0 0.0 0.0 [-2.775469, -7.3659234, 11.768431, 1.3434289, ... \n", + "4 0.0 0.0 0.0 0.0 0.0 [-4.942663, 4.0774136, -0.45292768, -0.9312788... \n", + "5 0.0 0.0 0.0 0.0 0.0 [-5.226616, -3.1389174, 2.6100307, 3.695045, -... \n", + "6 0.0 0.0 0.0 0.0 0.0 [-4.3006196, 5.116993, 0.5850617, -0.7624871, ... \n", + "7 0.0 0.0 0.0 0.0 0.0 [-2.398596, -1.4814726, -4.8840575, -0.2391601... \n", + "8 0.0 0.0 0.0 0.0 0.0 [0.82160157, -2.8640628, -1.6951559, -4.489291... \n", + "9 0.0 0.0 0.0 0.0 0.0 [-1.2338604, -2.151981, -4.1717424, 1.6106843,... \n", + "\n", + "[10 rows x 785 columns]" + ] + }, + "execution_count": 47, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "preds = df.withColumn(\"preds\", mnist(struct(df.columns))).limit(10).toPandas()\n", + "preds" + ] + }, + { + "cell_type": "code", + "execution_count": 48, + "id": "7c067c62-03a6-461e-a1ff-4653276fbea1", + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np" + ] + }, + { + "cell_type": "code", + "execution_count": 49, + "id": "a7084ad0-c021-4296-bad0-7a238971f53b", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "array([ -5.88436 , -3.1058552, 0.1087373, 12.67319 , -5.1437874,\n", + " 4.085992 , -10.203137 , -1.4333997, -3.3865087, -3.8473575],\n", + " dtype=float32)" + ] + }, + "execution_count": 49, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sample = preds.iloc[0]\n", + "sample.preds" + ] + }, + { + "cell_type": "code", + "execution_count": 50, + "id": "8167c832-93ef-4f50-873b-07b67c19ef53", + "metadata": {}, + "outputs": [], + "source": [ + "prediction = np.argmax(sample.preds)\n", + "img = sample.drop('preds').to_numpy(dtype=float)\n", + "img = np.array(img).reshape(28,28)" + ] + }, + { + "cell_type": "code", + "execution_count": 51, + "id": "297811e1-aecb-4afd-9a6a-30c49e8881cc", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAkfElEQVR4nO3dfXQUdZ7v8U/nqSEkaR7yLAFCFHRAcAYly/AgSiQEZUCYGUG9F7gziJiggI6KR0Udzsksrg7qIHjcHVhHEGWOyMoiDg9JGBRwwTCIM2QhJ0g4kIBcSYcAIaR/9w+uvbQkQDUdfkl4v86pc+iq37fqm6Lgk+qqrnYZY4wAALjKwmw3AAC4NhFAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAgAPdunXTpEmT/K8LCwvlcrlUWFgYsm24XC698MILIVsf0FwRQGgxlixZIpfL5Z/atGmjHj16KC8vT5WVlbbbc2TNmjUtJmTefvtt3X777UpKSpLb7VZ6eromT56s/fv3224NLVyE7QYAp1566SWlp6fr9OnT2rx5sxYuXKg1a9Zo9+7dio6Ovqq9DBkyRKdOnVJUVJSjujVr1mjBggUNhtCpU6cUEdF8/mkWFxcrPT1dP/vZz9ShQweVlZXp7bff1urVq/W3v/1NqamptltEC9V8jnLgMuXk5OjWW2+VJP36179Wp06d9Oqrr2rVqlWaMGFCgzU1NTVq165dyHsJCwtTmzZtQrrOUK/vSr355psXzBszZoxuvfVWvfPOO3r66actdIXWgLfg0OLdeeedkqSysjJJ0qRJkxQTE6PS0lKNHDlSsbGxeuCBByRJPp9P8+fPV69evdSmTRslJSVp6tSp+u677wLWaYzR3Llz1blzZ0VHR+uOO+7Q119/fcG2G7sGtG3bNo0cOVIdOnRQu3bt1KdPH7322mv+/hYsWCBJAW8pfq+ha0DFxcXKyclRXFycYmJiNGzYMG3dujVgzPdvUX722WeaNWuWEhIS1K5dO9177706evRowNiqqirt2bNHVVVVl7OLL9CtWzdJ0vHjx4OqByTOgNAKlJaWSpI6derkn3f27FllZ2dr0KBB+pd/+Rf/W3NTp07VkiVLNHnyZD366KMqKyvTH/7wBxUXF+uzzz5TZGSkJOn555/X3LlzNXLkSI0cOVJffvmlhg8frjNnzlyyn3Xr1umee+5RSkqKHnvsMSUnJ+sf//iHVq9erccee0xTp07VoUOHtG7dOv3pT3+65Pq+/vprDR48WHFxcXryyScVGRmpt956S0OHDlVRUZEyMzMDxk+fPl0dOnTQnDlztH//fs2fP195eXl6//33/WNWrlypyZMna/HixQE3VVzMsWPHVF9frwMHDuill16SJA0bNuyyaoEGGaCFWLx4sZFk1q9fb44ePWrKy8vN8uXLTadOnUzbtm3NwYMHjTHGTJw40UgyTz/9dED9X//6VyPJLF26NGD+2rVrA+YfOXLEREVFmbvvvtv4fD7/uGeeecZIMhMnTvTPKygoMJJMQUGBMcaYs2fPmvT0dNO1a1fz3XffBWzn/HXl5uaaxv75STJz5szxvx4zZoyJiooypaWl/nmHDh0ysbGxZsiQIRfsn6ysrIBtzZw504SHh5vjx49fMHbx4sUN9tAQt9ttJBlJplOnTub111+/7FqgIbwFhxYnKytLCQkJSktL0/jx4xUTE6OVK1fquuuuCxg3bdq0gNcrVqyQx+PRXXfdpW+//dY/9evXTzExMSooKJAkrV+/XmfOnNH06dMD3hqbMWPGJXsrLi5WWVmZZsyYofbt2wcsO39dl6u+vl5/+ctfNGbMGHXv3t0/PyUlRffff782b94sr9cbUPPQQw8FbGvw4MGqr6/XN9984583adIkGWMu++xHkj755BOtWbNGr7zyirp06aKamhrHPw9wPt6CQ4uzYMEC9ejRQxEREUpKSlLPnj0VFhb4u1RERIQ6d+4cMG/v3r2qqqpSYmJig+s9cuSIJPn/o77hhhsClickJKhDhw4X7e37twN79+59+T/QRRw9elQnT55Uz549L1h20003yefzqby8XL169fLP79KlS8C473v+4XUup+644w5J524CGT16tHr37q2YmBjl5eVd0Xpx7SKA0OL079/ffxdcY9xu9wWh5PP5lJiYqKVLlzZYk5CQELIebQoPD29wvjEmZNvIyMjQj3/8Yy1dupQAQtAIIFwzMjIytH79eg0cOFBt27ZtdFzXrl0lnTtjOv9tr6NHj17yLCIjI0OStHv3bmVlZTU67nLfjktISFB0dLRKSkouWLZnzx6FhYUpLS3tstYVaqdOnVJtba2VbaN14BoQrhm//OUvVV9fr9/+9rcXLDt79qz/luKsrCxFRkbqjTfeCDhrmD9//iW38ZOf/ETp6emaP3/+Bbcon7+u7z+TdKnbmMPDwzV8+HCtWrUq4MkDlZWVWrZsmQYNGqS4uLhL9vVDl3sb9tmzZxsM3S+++EJfffXVJc9EgYvhDAjXjNtvv11Tp05Vfn6+du7cqeHDhysyMlJ79+7VihUr9Nprr+nnP/+5EhIS9MQTTyg/P1/33HOPRo4cqeLiYn3yySeKj4+/6DbCwsK0cOFCjRo1SrfccosmT56slJQU7dmzR19//bU+/fRTSVK/fv0kSY8++qiys7MVHh6u8ePHN7jOuXPnat26dRo0aJAeeeQRRURE6K233lJtba3mzZsX1L643NuwT5w4obS0NN13333q1auX2rVrp6+++kqLFy+Wx+PRc889F9T2AYkAwjVm0aJF6tevn9566y0988wzioiIULdu3fTggw9q4MCB/nFz585VmzZttGjRIhUUFCgzM1N/+ctfdPfdd19yG9nZ2SooKNCLL76oV155RT6fTxkZGZoyZYp/zNixYzV9+nQtX75c7777rowxjQZQr1699Ne//lWzZ89Wfn6+fD6fMjMz9e67717wGaBQi46O1q9//WsVFBToz3/+s06dOqXU1FRNmDBBzz77rP8DqUAwXCaUVyYBALhMXAMCAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCKZvc5IJ/Pp0OHDik2NjaopwcDAOwyxqi6ulqpqakXPJPxfM0ugA4dOmTt2VYAgNApLy+/4Kn052t2ARQbGytJGqSRilCk5W4AAE6dVZ02a43///PGNFkALViwQC+//LIqKirUt29fvfHGG+rfv/8l675/2y1CkYpwEUAA0OL8/+frXOoySpPchPD+++9r1qxZmjNnjr788kv17dtX2dnZ/i/8AgCgSQLo1Vdf1ZQpUzR58mT96Ec/0qJFixQdHa0//vGPTbE5AEALFPIAOnPmjHbs2BHwZVxhYWHKysrSli1bLhhfW1srr9cbMAEAWr+QB9C3336r+vp6JSUlBcxPSkpSRUXFBePz8/Pl8Xj8E3fAAcC1wfoHUWfPnq2qqir/VF5ebrslAMBVEPK74OLj4xUeHq7KysqA+ZWVlUpOTr5gvNvtltvtDnUbAIBmLuRnQFFRUerXr582bNjgn+fz+bRhwwYNGDAg1JsDALRQTfI5oFmzZmnixIm69dZb1b9/f82fP181NTWaPHlyU2wOANACNUkA3XfffTp69Kief/55VVRU6JZbbtHatWsvuDEBAHDtchljjO0mzuf1euXxeDRUo3kSAgC0QGdNnQq1SlVVVYqLi2t0nPW74AAA1yYCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWBFhuwHYZ37aN7i6cOe/v0RWeh3XlP7vRMc1vu6nHNdI0p7b/+i4JtzlfD9MPTjAcU3hp7c4run2nzWOayRJW3cFVwc4wBkQAMAKAggAYEXIA+iFF16Qy+UKmG688cZQbwYA0MI1yTWgXr16af369f+zkQguNQEAAjVJMkRERCg5ObkpVg0AaCWa5BrQ3r17lZqaqu7du+uBBx7QgQMHGh1bW1srr9cbMAEAWr+QB1BmZqaWLFmitWvXauHChSorK9PgwYNVXV3d4Pj8/Hx5PB7/lJaWFuqWAADNUMgDKCcnR7/4xS/Up08fZWdna82aNTp+/Lg++OCDBsfPnj1bVVVV/qm8vDzULQEAmqEmvzugffv26tGjh/bt29fgcrfbLbfb3dRtAACamSb/HNCJEydUWlqqlJSUpt4UAKAFCXkAPfHEEyoqKtL+/fv1+eef695771V4eLgmTJgQ6k0BAFqwkL8Fd/DgQU2YMEHHjh1TQkKCBg0apK1btyohISHUmwIAtGAuY4yx3cT5vF6vPB6Phmq0IlyRttuxqubnmY5rKm91flK7dsLLjmskqUtEW8c1/2v/XY5r/tRtneManFN8xhdU3eOP5zmuif5wW1DbQutz1tSpUKtUVVWluLi4RsfxLDgAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIKHkV4lR/J+6rim8OlXHNdEu6Ic1zR339afclzTxhXc71Z1cv7PYcaBexzX/DLxvxzX3B1d5bgmWPvqah3XPDH4l45rzpYfdFyD5o+HkQIAmjUCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsiLDdwLXCF+68pjU+2frlYz9yXLNhxiDHNfVtg/vd6rvrnT+B/br/POy45s2EcY5r7v7zHx3XBGvsf011XNPt+P7QN4JWjTMgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCh5FeJan/+jfHNR88kui4Jjv6gOOanDlPOK6RpLoYl+Oa6/7joOOaiP07nNc4rjgnOYia+iBqKu/5aRBVV8+uny5xXDMmiAes+qqrHdeg9eAMCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCs4GGkV4mvpsZxzTs90xzXvJ0z1nFNfEGx4xpJ8p0+7bjmbFBbunrCExIc13x3V4bjmscf/sBxDdDacAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcByY4yef/55paSkqG3btsrKytLevXtD1S8AoJVwHEA1NTXq27evFixY0ODyefPm6fXXX9eiRYu0bds2tWvXTtnZ2TodxPUCAEDr5fgmhJycHOXk5DS4zBij+fPn69lnn9Xo0aMlSe+8846SkpL00Ucfafz48VfWLQCg1QjpNaCysjJVVFQoKyvLP8/j8SgzM1NbtmxpsKa2tlZerzdgAgC0fiENoIqKCklSUlJSwPykpCT/sh/Kz8+Xx+PxT2lpzm89BgC0PNbvgps9e7aqqqr8U3l5ue2WAABXQUgDKDk5WZJUWVkZML+ystK/7Ifcbrfi4uICJgBA6xfSAEpPT1dycrI2bNjgn+f1erVt2zYNGDAglJsCALRwju+CO3HihPbt2+d/XVZWpp07d6pjx47q0qWLZsyYoblz5+qGG25Qenq6nnvuOaWmpmrMmDGh7BsA0MI5DqDt27frjjvu8L+eNWuWJGnixIlasmSJnnzySdXU1Oihhx7S8ePHNWjQIK1du1Zt2rQJXdcAgBbPZYwxtps4n9frlcfj0VCNVoQr0nY7aKHC23uCqnt8x2bHNUPanAlqW1eDT76g6n73bV/HNduGpTiuqf/2mOMaNH9nTZ0KtUpVVVUXva5v/S44AMC1iQACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACscfx0D0BKUPdorqLohbTaGuBO7VtXEB1X3ed+oIKp4sjWc4QwIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKzgYaRAK3ZX28NB1c2d9YDjmrqYoDblWGLxWcc1bT7+ogk6wZXiDAgAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArOBhpGiV0jacDKpuxyTnNf3cQW3qqogJC665HY+/EeJOQmfOkR87rtnxMb9rN0f8rQAArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFTyMFK2S67OdQdW9NOp+xzW1STGOa6of9zqu+eyW5Y5rWqNnE7Y7rrnzwUeD2pbn3a1B1eHycAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcDySZMmyeVyBUwjRowIVb8AgFbCcQDV1NSob9++WrBgQaNjRowYocOHD/un995774qaBAC0Po5vQsjJyVFOTs5Fx7jdbiUnJwfdFACg9WuSa0CFhYVKTExUz549NW3aNB07dqzRsbW1tfJ6vQETAKD1C3kAjRgxQu+88442bNigf/7nf1ZRUZFycnJUX1/f4Pj8/Hx5PB7/lJaWFuqWAADNUMg/BzR+/Hj/n2+++Wb16dNHGRkZKiws1LBhwy4YP3v2bM2aNcv/2uv1EkIAcA1o8tuwu3fvrvj4eO3bt6/B5W63W3FxcQETAKD1a/IAOnjwoI4dO6aUlJSm3hQAoAVx/BbciRMnAs5mysrKtHPnTnXs2FEdO3bUiy++qHHjxik5OVmlpaV68skndf311ys7OzukjQMAWjbHAbR9+3bdcccd/tffX7+ZOHGiFi5cqF27dunf//3fdfz4caWmpmr48OH67W9/K7fbHbquAQAtnuMAGjp0qIwxjS7/9NNPr6ghwKb6r0sc10R87Xw7HQpcjmtGRf3Ucc3+P/VwXCNJn2QudFzTOaJtUNtyKtIV7rjmdMfgrjZ4gqrC5eJZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAi5F/JDeAyXOSJ8o2W1NY6run6y68c10jSnW/NdFzz3/csCmpbuHZxBgQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVvAwUqAVc0VGBVfXtj7EnYTOrjPOe0vcXtMEneBKcQYEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFbwMFKgFSt545ag6v572MLQNhJCMx6f7rgm+vNtTdAJrhRnQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQ8jRasUFhsbXF17T4g7adiRu9Ic19w1/TPHNf+RuMBxzTlX53fTD04kOq6J2/KN45qzjitwNXAGBACwggACAFjhKIDy8/N12223KTY2VomJiRozZoxKSkoCxpw+fVq5ubnq1KmTYmJiNG7cOFVWVoa0aQBAy+cogIqKipSbm6utW7dq3bp1qqur0/Dhw1VTU+MfM3PmTH388cdasWKFioqKdOjQIY0dOzbkjQMAWjZHNyGsXbs24PWSJUuUmJioHTt2aMiQIaqqqtK//du/admyZbrzzjslSYsXL9ZNN92krVu36p/+6Z9C1zkAoEW7omtAVVVVkqSOHTtKknbs2KG6ujplZWX5x9x4443q0qWLtmzZ0uA6amtr5fV6AyYAQOsXdAD5fD7NmDFDAwcOVO/evSVJFRUVioqKUvv27QPGJiUlqaKiosH15Ofny+Px+Ke0NOe3pwIAWp6gAyg3N1e7d+/W8uXLr6iB2bNnq6qqyj+Vl5df0foAAC1DUB9EzcvL0+rVq7Vp0yZ17tzZPz85OVlnzpzR8ePHA86CKisrlZyc3OC63G633G53MG0AAFowR2dAxhjl5eVp5cqV2rhxo9LT0wOW9+vXT5GRkdqwYYN/XklJiQ4cOKABAwaEpmMAQKvg6AwoNzdXy5Yt06pVqxQbG+u/ruPxeNS2bVt5PB796le/0qxZs9SxY0fFxcVp+vTpGjBgAHfAAQACOAqghQsXSpKGDh0aMH/x4sWaNGmSJOn3v/+9wsLCNG7cONXW1io7O1tvvvlmSJoFALQeLmOMsd3E+bxerzwej4ZqtCJckbbbuSaE9b0pqLo9uTGOa5LT/q/jmiMlCY5rJt9Z6LhGkp7q9HVQdQhOn88nOa7p8ouvQt8IQuqsqVOhVqmqqkpxcXGNjuNZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAiqG9ERfPl6tfLcU3b3x8Jalv/nfFuUHWO9bk6m2nuak2d45pIV3hQ26qsr3VcM+dQjuOazq8F1x9aB86AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKHkbaytR0jXFc8373fw1ya1FB1jU9n3xB1c08NNhxzW8S1zuuyf4813FNbGG045rqbo5LJEnps7cEUVXtuCJMO4PYDloLzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoeRtrKRH+4zXHNLw4/HNS2jv64neMaXxDPL61z/nxVvf1//uC8SFLpbacd10z78VTHNek7dzmukTGOS+KdbwW4ajgDAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArXMYE8YTDJuT1euXxeDRUoxXhirTdDgDAobOmToVapaqqKsXFxTU6jjMgAIAVBBAAwApHAZSfn6/bbrtNsbGxSkxM1JgxY1RSUhIwZujQoXK5XAHTww8H930zAIDWy1EAFRUVKTc3V1u3btW6detUV1en4cOHq6amJmDclClTdPjwYf80b968kDYNAGj5HH0j6tq1awNeL1myRImJidqxY4eGDBninx8dHa3k5OTQdAgAaJWu6BpQVVWVJKljx44B85cuXar4+Hj17t1bs2fP1smTJxtdR21trbxeb8AEAGj9HJ0Bnc/n82nGjBkaOHCgevfu7Z9///33q2vXrkpNTdWuXbv01FNPqaSkRB9++GGD68nPz9eLL74YbBsAgBYq6M8BTZs2TZ988ok2b96szp07Nzpu48aNGjZsmPbt26eMjIwLltfW1qq2ttb/2uv1Ki0tjc8BAUALdbmfAwrqDCgvL0+rV6/Wpk2bLho+kpSZmSlJjQaQ2+2W2+0Opg0AQAvmKICMMZo+fbpWrlypwsJCpaenX7Jm586dkqSUlJSgGgQAtE6OAig3N1fLli3TqlWrFBsbq4qKCkmSx+NR27ZtVVpaqmXLlmnkyJHq1KmTdu3apZkzZ2rIkCHq06dPk/wAAICWydE1IJfL1eD8xYsXa9KkSSovL9eDDz6o3bt3q6amRmlpabr33nv17LPPXvR9wPPxLDgAaNma5BrQpbIqLS1NRUVFTlYJALhG8Sw4AIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVEbYb+CFjjCTprOokY7kZAIBjZ1Un6X/+P29Mswug6upqSdJmrbHcCQDgSlRXV8vj8TS63GUuFVFXmc/n06FDhxQbGyuXyxWwzOv1Ki0tTeXl5YqLi7PUoX3sh3PYD+ewH85hP5zTHPaDMUbV1dVKTU1VWFjjV3qa3RlQWFiYOnfufNExcXFx1/QB9j32wznsh3PYD+ewH86xvR8udubzPW5CAABYQQABAKxoUQHkdrs1Z84cud1u261YxX44h/1wDvvhHPbDOS1pPzS7mxAAANeGFnUGBABoPQggAIAVBBAAwAoCCABgBQEEALCixQTQggUL1K1bN7Vp00aZmZn64osvbLd01b3wwgtyuVwB04033mi7rSa3adMmjRo1SqmpqXK5XProo48Clhtj9PzzzyslJUVt27ZVVlaW9u7da6fZJnSp/TBp0qQLjo8RI0bYabaJ5Ofn67bbblNsbKwSExM1ZswYlZSUBIw5ffq0cnNz1alTJ8XExGjcuHGqrKy01HHTuJz9MHTo0AuOh4cffthSxw1rEQH0/vvva9asWZozZ46+/PJL9e3bV9nZ2Tpy5Ijt1q66Xr166fDhw/5p8+bNtltqcjU1Nerbt68WLFjQ4PJ58+bp9ddf16JFi7Rt2za1a9dO2dnZOn369FXutGldaj9I0ogRIwKOj/fee+8qdtj0ioqKlJubq61bt2rdunWqq6vT8OHDVVNT4x8zc+ZMffzxx1qxYoWKiop06NAhjR071mLXoXc5+0GSpkyZEnA8zJs3z1LHjTAtQP/+/U1ubq7/dX19vUlNTTX5+fkWu7r65syZY/r27Wu7DaskmZUrV/pf+3w+k5ycbF5++WX/vOPHjxu3223ee+89Cx1eHT/cD8YYM3HiRDN69Ggr/dhy5MgRI8kUFRUZY8793UdGRpoVK1b4x/zjH/8wksyWLVtstdnkfrgfjDHm9ttvN4899pi9pi5Dsz8DOnPmjHbs2KGsrCz/vLCwMGVlZWnLli0WO7Nj7969Sk1NVffu3fXAAw/owIEDtluyqqysTBUVFQHHh8fjUWZm5jV5fBQWFioxMVE9e/bUtGnTdOzYMdstNamqqipJUseOHSVJO3bsUF1dXcDxcOONN6pLly6t+nj44X743tKlSxUfH6/evXtr9uzZOnnypI32GtXsnob9Q99++63q6+uVlJQUMD8pKUl79uyx1JUdmZmZWrJkiXr27KnDhw/rxRdf1ODBg7V7927Fxsbabs+KiooKSWrw+Ph+2bVixIgRGjt2rNLT01VaWqpnnnlGOTk52rJli8LDw223F3I+n08zZszQwIED1bt3b0nnjoeoqCi1b98+YGxrPh4a2g+SdP/996tr165KTU3Vrl279NRTT6mkpEQffvihxW4DNfsAwv/Iycnx/7lPnz7KzMxU165d9cEHH+hXv/qVxc7QHIwfP97/55tvvll9+vRRRkaGCgsLNWzYMIudNY3c3Fzt3r37mrgOejGN7YeHHnrI/+ebb75ZKSkpGjZsmEpLS5WRkXG122xQs38LLj4+XuHh4RfcxVJZWank5GRLXTUP7du3V48ePbRv3z7brVjz/THA8XGh7t27Kz4+vlUeH3l5eVq9erUKCgoCvj8sOTlZZ86c0fHjxwPGt9bjobH90JDMzExJalbHQ7MPoKioKPXr108bNmzwz/P5fNqwYYMGDBhgsTP7Tpw4odLSUqWkpNhuxZr09HQlJycHHB9er1fbtm275o+PgwcP6tixY63q+DDGKC8vTytXrtTGjRuVnp4esLxfv36KjIwMOB5KSkp04MCBVnU8XGo/NGTnzp2S1LyOB9t3QVyO5cuXG7fbbZYsWWL+/ve/m4ceesi0b9/eVFRU2G7tqnr88cdNYWGhKSsrM5999pnJysoy8fHx5siRI7Zba1LV1dWmuLjYFBcXG0nm1VdfNcXFxeabb74xxhjzu9/9zrRv396sWrXK7Nq1y4wePdqkp6ebU6dOWe48tC62H6qrq80TTzxhtmzZYsrKysz69evNT37yE3PDDTeY06dP2249ZKZNm2Y8Ho8pLCw0hw8f9k8nT570j3n44YdNly5dzMaNG8327dvNgAEDzIABAyx2HXqX2g/79u0zL730ktm+fbspKyszq1atMt27dzdDhgyx3HmgFhFAxhjzxhtvmC5dupioqCjTv39/s3XrVtstXXX33XefSUlJMVFRUea6664z9913n9m3b5/ttppcQUGBkXTBNHHiRGPMuVuxn3vuOZOUlGTcbrcZNmyYKSkpsdt0E7jYfjh58qQZPny4SUhIMJGRkaZr165mypQpre6XtIZ+fklm8eLF/jGnTp0yjzzyiOnQoYOJjo429957rzl8+LC9ppvApfbDgQMHzJAhQ0zHjh2N2+02119/vfnNb35jqqqq7Db+A3wfEADAimZ/DQgA0DoRQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAV/w/hgVLrpVGHsAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.figure()\n", + "plt.title(\"Prediction: {}\".format(prediction))\n", + "plt.imshow(img)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "5961593d-182e-4620-9a5e-f98ba3d2534d", + "metadata": {}, + "source": [ + "### Using Triton Inference Server\n", + "\n", + "Note: you can restart the kernel and run from this point to simulate running in a different node or environment." + ] + }, + { + "cell_type": "code", + "execution_count": 52, + "id": "a64d19b1-ba4a-4dc7-b3a9-368dc47d0fd8", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import col, struct\n", + "from pyspark.sql.types import ArrayType, FloatType" + ] + }, + { + "cell_type": "code", + "execution_count": 53, + "id": "8fa92fe4-2e04-4d82-a357-bfdfca38bd8c", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "# copy model to expected layout for Triton\n", + "rm -rf models\n", + "mkdir -p models/mnist_model/1\n", + "cp -r mnist_model models/mnist_model/1/model.savedmodel\n", + "\n", + "# add config.pbtxt\n", + "cp models_config/mnist_model/config.pbtxt models/mnist_model/config.pbtxt" + ] + }, + { + "cell_type": "markdown", + "id": "f1673e0e-5c75-44e1-88c6-5f5cf1275e4b", + "metadata": {}, + "source": [ + "#### Start Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": 54, + "id": "0f7ecb25-be16-40c4-bdbb-441e2f537000", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "[True]" + ] + }, + "execution_count": 54, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "num_executors = 1\n", + "triton_models_dir = \"{}/models\".format(os.getcwd())\n", + "nodeRDD = sc.parallelize(list(range(num_executors)), num_executors)\n", + "\n", + "def start_triton(it):\n", + " import docker\n", + " import time\n", + " import tritonclient.grpc as grpcclient\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " if containers:\n", + " \n", + " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", + " else:\n", + " container=client.containers.run(\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", + " detach=True,\n", + " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", + " name=\"spark-triton\",\n", + " network_mode=\"host\",\n", + " remove=True,\n", + " shm_size=\"64M\",\n", + " volumes={triton_models_dir: {\"bind\": \"/models\", \"mode\": \"ro\"}}\n", + " )\n", + " print(\">>>> starting triton: {}\".format(container.short_id))\n", + "\n", + " # wait for triton to be running\n", + " time.sleep(15)\n", + " client = grpcclient.InferenceServerClient(\"localhost:8001\")\n", + " ready = False\n", + " while not ready:\n", + " try:\n", + " ready = client.is_server_ready()\n", + " except Exception as e:\n", + " time.sleep(5)\n", + " \n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(start_triton).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 55, + "id": "43b93753-1d52-4060-9986-f24c30a67528", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "StructType([StructField('data', ArrayType(DoubleType(), True), True)])" + ] + }, + "execution_count": 55, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "df = spark.read.parquet(\"mnist_1\")\n", + "df.schema" + ] + }, + { + "cell_type": "markdown", + "id": "036680eb-babd-4b07-8b2c-ce6e724f4e85", + "metadata": {}, + "source": [ + "#### Run inference" + ] + }, + { + "cell_type": "code", + "execution_count": 56, + "id": "3af08bd0-3838-4769-a8de-2643db4101c6", + "metadata": {}, + "outputs": [], + "source": [ + "def triton_fn(triton_uri, model_name):\n", + " import numpy as np\n", + " import tritonclient.grpc as grpcclient\n", + "\n", + " np_types = {\n", + " \"BOOL\": np.dtype(np.bool_),\n", + " \"INT8\": np.dtype(np.int8),\n", + " \"INT16\": np.dtype(np.int16),\n", + " \"INT32\": np.dtype(np.int32),\n", + " \"INT64\": np.dtype(np.int64),\n", + " \"FP16\": np.dtype(np.float16),\n", + " \"FP32\": np.dtype(np.float32),\n", + " \"FP64\": np.dtype(np.float64),\n", + " \"FP64\": np.dtype(np.double),\n", + " \"BYTES\": np.dtype(object)\n", + " }\n", + "\n", + " client = grpcclient.InferenceServerClient(triton_uri)\n", + " model_meta = client.get_model_metadata(model_name)\n", + "\n", + " def predict(inputs):\n", + " if isinstance(inputs, np.ndarray):\n", + " # single ndarray input\n", + " request = [grpcclient.InferInput(model_meta.inputs[0].name, inputs.shape, model_meta.inputs[0].datatype)]\n", + " request[0].set_data_from_numpy(inputs.astype(np_types[model_meta.inputs[0].datatype]))\n", + " else:\n", + " # dict of multiple ndarray inputs\n", + " request = [grpcclient.InferInput(i.name, inputs[i.name].shape, i.datatype) for i in model_meta.inputs]\n", + " for i in request:\n", + " i.set_data_from_numpy(inputs[i.name()].astype(np_types[i.datatype()]))\n", + "\n", + " response = client.infer(model_name, inputs=request)\n", + "\n", + " if len(model_meta.outputs) > 1:\n", + " # return dictionary of numpy arrays\n", + " return {o.name: response.as_numpy(o.name) for o in model_meta.outputs}\n", + " else:\n", + " # return single numpy array\n", + " return response.as_numpy(model_meta.outputs[0].name)\n", + "\n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 57, + "id": "6658d2a1-ef7b-4ca1-9fb6-f2ac9050f3e5", + "metadata": {}, + "outputs": [], + "source": [ + "from functools import partial\n", + "\n", + "predict = predict_batch_udf(partial(triton_fn, \"localhost:8001\", \"mnist_model\"),\n", + " return_type=ArrayType(FloatType()),\n", + " input_tensor_shapes=[[784]],\n", + " batch_size=8192)" + ] + }, + { + "cell_type": "code", + "execution_count": 58, + "id": "8397aa14-82fd-4351-a477-dc8e8b321fa2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 20.1 ms, sys: 3.41 ms, total: 23.5 ms\n", + "Wall time: 625 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", predict(struct(\"data\"))).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 59, + "id": "82698bd9-377a-4415-8971-835487f876cc", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 30.3 ms, sys: 8.81 ms, total: 39.2 ms\n", + "Wall time: 154 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", predict(\"data\")).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 60, + "id": "419ad7bd-fa28-49d3-b98d-db9fba5aeaef", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 2.67 ms, sys: 4.2 ms, total: 6.87 ms\n", + "Wall time: 131 ms\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
datapreds
0[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-5.7614846, -3.52228, -1.1202906, 13.053683, ...
1[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-3.1390061, -8.71185, 0.82955813, -4.034869, ...
2[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-3.046528, 0.3521706, 0.6788677, 0.72303534, ...
3[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-2.401024, -7.6780066, 11.145876, 1.2857256, ...
4[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-5.0012593, 3.806796, -0.8154834, -0.9550028,...
5[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-5.0425925, -3.4815094, 1.641246, 3.608149, -...
6[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-4.288771, 5.0072904, 0.27649477, -0.797148, ...
7[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-2.2032878, -1.6879876, -5.874276, -0.5945335...
8[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[1.1337761, -3.1751056, -2.5246286, -5.028277,...
9[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ...[-0.92484117, -2.4703276, -5.023897, 1.46669, ...
\n", + "
" + ], + "text/plain": [ + " data \\\n", + "0 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "1 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "2 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "3 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "4 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "5 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "6 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "7 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "8 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "9 [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, ... \n", + "\n", + " preds \n", + "0 [-5.7614846, -3.52228, -1.1202906, 13.053683, ... \n", + "1 [-3.1390061, -8.71185, 0.82955813, -4.034869, ... \n", + "2 [-3.046528, 0.3521706, 0.6788677, 0.72303534, ... \n", + "3 [-2.401024, -7.6780066, 11.145876, 1.2857256, ... \n", + "4 [-5.0012593, 3.806796, -0.8154834, -0.9550028,... \n", + "5 [-5.0425925, -3.4815094, 1.641246, 3.608149, -... \n", + "6 [-4.288771, 5.0072904, 0.27649477, -0.797148, ... \n", + "7 [-2.2032878, -1.6879876, -5.874276, -0.5945335... \n", + "8 [1.1337761, -3.1751056, -2.5246286, -5.028277,... \n", + "9 [-0.92484117, -2.4703276, -5.023897, 1.46669, ... " + ] + }, + "execution_count": 60, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "%%time\n", + "preds = df.withColumn(\"preds\", predict(col(\"data\"))).limit(10).toPandas()\n", + "preds" + ] + }, + { + "cell_type": "code", + "execution_count": 61, + "id": "79d90a26", + "metadata": {}, + "outputs": [], + "source": [ + "import matplotlib.pyplot as plt\n", + "import numpy as np" + ] + }, + { + "cell_type": "code", + "execution_count": 62, + "id": "4ca495f5", + "metadata": {}, + "outputs": [], + "source": [ + "sample = preds.iloc[0]\n", + "sample.preds\n", + "\n", + "prediction = np.argmax(sample.preds)\n", + "img = np.array(sample.data).reshape(28,28)" + ] + }, + { + "cell_type": "code", + "execution_count": 63, + "id": "a5d10903", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAaAAAAGzCAYAAABpdMNsAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAkfElEQVR4nO3dfXQUdZ7v8U/nqSEkaR7yLAFCFHRAcAYly/AgSiQEZUCYGUG9F7gziJiggI6KR0Udzsksrg7qIHjcHVhHEGWOyMoiDg9JGBRwwTCIM2QhJ0g4kIBcSYcAIaR/9w+uvbQkQDUdfkl4v86pc+iq37fqm6Lgk+qqrnYZY4wAALjKwmw3AAC4NhFAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAgAPdunXTpEmT/K8LCwvlcrlUWFgYsm24XC698MILIVsf0FwRQGgxlixZIpfL5Z/atGmjHj16KC8vT5WVlbbbc2TNmjUtJmTefvtt3X777UpKSpLb7VZ6eromT56s/fv3224NLVyE7QYAp1566SWlp6fr9OnT2rx5sxYuXKg1a9Zo9+7dio6Ovqq9DBkyRKdOnVJUVJSjujVr1mjBggUNhtCpU6cUEdF8/mkWFxcrPT1dP/vZz9ShQweVlZXp7bff1urVq/W3v/1NqamptltEC9V8jnLgMuXk5OjWW2+VJP36179Wp06d9Oqrr2rVqlWaMGFCgzU1NTVq165dyHsJCwtTmzZtQrrOUK/vSr355psXzBszZoxuvfVWvfPOO3r66actdIXWgLfg0OLdeeedkqSysjJJ0qRJkxQTE6PS0lKNHDlSsbGxeuCBByRJPp9P8+fPV69evdSmTRslJSVp6tSp+u677wLWaYzR3Llz1blzZ0VHR+uOO+7Q119/fcG2G7sGtG3bNo0cOVIdOnRQu3bt1KdPH7322mv+/hYsWCBJAW8pfq+ha0DFxcXKyclRXFycYmJiNGzYMG3dujVgzPdvUX722WeaNWuWEhIS1K5dO9177706evRowNiqqirt2bNHVVVVl7OLL9CtWzdJ0vHjx4OqByTOgNAKlJaWSpI6derkn3f27FllZ2dr0KBB+pd/+Rf/W3NTp07VkiVLNHnyZD366KMqKyvTH/7wBxUXF+uzzz5TZGSkJOn555/X3LlzNXLkSI0cOVJffvmlhg8frjNnzlyyn3Xr1umee+5RSkqKHnvsMSUnJ+sf//iHVq9erccee0xTp07VoUOHtG7dOv3pT3+65Pq+/vprDR48WHFxcXryyScVGRmpt956S0OHDlVRUZEyMzMDxk+fPl0dOnTQnDlztH//fs2fP195eXl6//33/WNWrlypyZMna/HixQE3VVzMsWPHVF9frwMHDuill16SJA0bNuyyaoEGGaCFWLx4sZFk1q9fb44ePWrKy8vN8uXLTadOnUzbtm3NwYMHjTHGTJw40UgyTz/9dED9X//6VyPJLF26NGD+2rVrA+YfOXLEREVFmbvvvtv4fD7/uGeeecZIMhMnTvTPKygoMJJMQUGBMcaYs2fPmvT0dNO1a1fz3XffBWzn/HXl5uaaxv75STJz5szxvx4zZoyJiooypaWl/nmHDh0ysbGxZsiQIRfsn6ysrIBtzZw504SHh5vjx49fMHbx4sUN9tAQt9ttJBlJplOnTub111+/7FqgIbwFhxYnKytLCQkJSktL0/jx4xUTE6OVK1fquuuuCxg3bdq0gNcrVqyQx+PRXXfdpW+//dY/9evXTzExMSooKJAkrV+/XmfOnNH06dMD3hqbMWPGJXsrLi5WWVmZZsyYofbt2wcsO39dl6u+vl5/+ctfNGbMGHXv3t0/PyUlRffff782b94sr9cbUPPQQw8FbGvw4MGqr6/XN9984583adIkGWMu++xHkj755BOtWbNGr7zyirp06aKamhrHPw9wPt6CQ4uzYMEC9ejRQxEREUpKSlLPnj0VFhb4u1RERIQ6d+4cMG/v3r2qqqpSYmJig+s9cuSIJPn/o77hhhsClickJKhDhw4X7e37twN79+59+T/QRRw9elQnT55Uz549L1h20003yefzqby8XL169fLP79KlS8C473v+4XUup+644w5J524CGT16tHr37q2YmBjl5eVd0Xpx7SKA0OL079/ffxdcY9xu9wWh5PP5lJiYqKVLlzZYk5CQELIebQoPD29wvjEmZNvIyMjQj3/8Yy1dupQAQtAIIFwzMjIytH79eg0cOFBt27ZtdFzXrl0lnTtjOv9tr6NHj17yLCIjI0OStHv3bmVlZTU67nLfjktISFB0dLRKSkouWLZnzx6FhYUpLS3tstYVaqdOnVJtba2VbaN14BoQrhm//OUvVV9fr9/+9rcXLDt79qz/luKsrCxFRkbqjTfeCDhrmD9//iW38ZOf/ETp6emaP3/+Bbcon7+u7z+TdKnbmMPDwzV8+HCtWrUq4MkDlZWVWrZsmQYNGqS4uLhL9vVDl3sb9tmzZxsM3S+++EJfffXVJc9EgYvhDAjXjNtvv11Tp05Vfn6+du7cqeHDhysyMlJ79+7VihUr9Nprr+nnP/+5EhIS9MQTTyg/P1/33HOPRo4cqeLiYn3yySeKj4+/6DbCwsK0cOFCjRo1SrfccosmT56slJQU7dmzR19//bU+/fRTSVK/fv0kSY8++qiys7MVHh6u8ePHN7jOuXPnat26dRo0aJAeeeQRRURE6K233lJtba3mzZsX1L643NuwT5w4obS0NN13333q1auX2rVrp6+++kqLFy+Wx+PRc889F9T2AYkAwjVm0aJF6tevn9566y0988wzioiIULdu3fTggw9q4MCB/nFz585VmzZttGjRIhUUFCgzM1N/+ctfdPfdd19yG9nZ2SooKNCLL76oV155RT6fTxkZGZoyZYp/zNixYzV9+nQtX75c7777rowxjQZQr1699Ne//lWzZ89Wfn6+fD6fMjMz9e67717wGaBQi46O1q9//WsVFBToz3/+s06dOqXU1FRNmDBBzz77rP8DqUAwXCaUVyYBALhMXAMCAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMCKZvc5IJ/Pp0OHDik2NjaopwcDAOwyxqi6ulqpqakXPJPxfM0ugA4dOmTt2VYAgNApLy+/4Kn052t2ARQbGytJGqSRilCk5W4AAE6dVZ02a43///PGNFkALViwQC+//LIqKirUt29fvfHGG+rfv/8l675/2y1CkYpwEUAA0OL8/+frXOoySpPchPD+++9r1qxZmjNnjr788kv17dtX2dnZ/i/8AgCgSQLo1Vdf1ZQpUzR58mT96Ec/0qJFixQdHa0//vGPTbE5AEALFPIAOnPmjHbs2BHwZVxhYWHKysrSli1bLhhfW1srr9cbMAEAWr+QB9C3336r+vp6JSUlBcxPSkpSRUXFBePz8/Pl8Xj8E3fAAcC1wfoHUWfPnq2qqir/VF5ebrslAMBVEPK74OLj4xUeHq7KysqA+ZWVlUpOTr5gvNvtltvtDnUbAIBmLuRnQFFRUerXr582bNjgn+fz+bRhwwYNGDAg1JsDALRQTfI5oFmzZmnixIm69dZb1b9/f82fP181NTWaPHlyU2wOANACNUkA3XfffTp69Kief/55VVRU6JZbbtHatWsvuDEBAHDtchljjO0mzuf1euXxeDRUo3kSAgC0QGdNnQq1SlVVVYqLi2t0nPW74AAA1yYCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWBFhuwHYZ37aN7i6cOe/v0RWeh3XlP7vRMc1vu6nHNdI0p7b/+i4JtzlfD9MPTjAcU3hp7c4run2nzWOayRJW3cFVwc4wBkQAMAKAggAYEXIA+iFF16Qy+UKmG688cZQbwYA0MI1yTWgXr16af369f+zkQguNQEAAjVJMkRERCg5ObkpVg0AaCWa5BrQ3r17lZqaqu7du+uBBx7QgQMHGh1bW1srr9cbMAEAWr+QB1BmZqaWLFmitWvXauHChSorK9PgwYNVXV3d4Pj8/Hx5PB7/lJaWFuqWAADNUMgDKCcnR7/4xS/Up08fZWdna82aNTp+/Lg++OCDBsfPnj1bVVVV/qm8vDzULQEAmqEmvzugffv26tGjh/bt29fgcrfbLbfb3dRtAACamSb/HNCJEydUWlqqlJSUpt4UAKAFCXkAPfHEEyoqKtL+/fv1+eef695771V4eLgmTJgQ6k0BAFqwkL8Fd/DgQU2YMEHHjh1TQkKCBg0apK1btyohISHUmwIAtGAuY4yx3cT5vF6vPB6Phmq0IlyRttuxqubnmY5rKm91flK7dsLLjmskqUtEW8c1/2v/XY5r/tRtneManFN8xhdU3eOP5zmuif5wW1DbQutz1tSpUKtUVVWluLi4RsfxLDgAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIKHkV4lR/J+6rim8OlXHNdEu6Ic1zR339afclzTxhXc71Z1cv7PYcaBexzX/DLxvxzX3B1d5bgmWPvqah3XPDH4l45rzpYfdFyD5o+HkQIAmjUCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsiLDdwLXCF+68pjU+2frlYz9yXLNhxiDHNfVtg/vd6rvrnT+B/br/POy45s2EcY5r7v7zHx3XBGvsf011XNPt+P7QN4JWjTMgAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCh5FeJan/+jfHNR88kui4Jjv6gOOanDlPOK6RpLoYl+Oa6/7joOOaiP07nNc4rjgnOYia+iBqKu/5aRBVV8+uny5xXDMmiAes+qqrHdeg9eAMCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCs4GGkV4mvpsZxzTs90xzXvJ0z1nFNfEGx4xpJ8p0+7bjmbFBbunrCExIc13x3V4bjmscf/sBxDdDacAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcByY4yef/55paSkqG3btsrKytLevXtD1S8AoJVwHEA1NTXq27evFixY0ODyefPm6fXXX9eiRYu0bds2tWvXTtnZ2TodxPUCAEDr5fgmhJycHOXk5DS4zBij+fPn69lnn9Xo0aMlSe+8846SkpL00Ucfafz48VfWLQCg1QjpNaCysjJVVFQoKyvLP8/j8SgzM1NbtmxpsKa2tlZerzdgAgC0fiENoIqKCklSUlJSwPykpCT/sh/Kz8+Xx+PxT2lpzm89BgC0PNbvgps9e7aqqqr8U3l5ue2WAABXQUgDKDk5WZJUWVkZML+ystK/7Ifcbrfi4uICJgBA6xfSAEpPT1dycrI2bNjgn+f1erVt2zYNGDAglJsCALRwju+CO3HihPbt2+d/XVZWpp07d6pjx47q0qWLZsyYoblz5+qGG25Qenq6nnvuOaWmpmrMmDGh7BsA0MI5DqDt27frjjvu8L+eNWuWJGnixIlasmSJnnzySdXU1Oihhx7S8ePHNWjQIK1du1Zt2rQJXdcAgBbPZYwxtps4n9frlcfj0VCNVoQr0nY7aKHC23uCqnt8x2bHNUPanAlqW1eDT76g6n73bV/HNduGpTiuqf/2mOMaNH9nTZ0KtUpVVVUXva5v/S44AMC1iQACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACscfx0D0BKUPdorqLohbTaGuBO7VtXEB1X3ed+oIKp4sjWc4QwIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKzgYaRAK3ZX28NB1c2d9YDjmrqYoDblWGLxWcc1bT7+ogk6wZXiDAgAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArOBhpGiV0jacDKpuxyTnNf3cQW3qqogJC665HY+/EeJOQmfOkR87rtnxMb9rN0f8rQAArCCAAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFTyMFK2S67OdQdW9NOp+xzW1STGOa6of9zqu+eyW5Y5rWqNnE7Y7rrnzwUeD2pbn3a1B1eHycAYEALCCAAIAWOE4gDZt2qRRo0YpNTVVLpdLH330UcDySZMmyeVyBUwjRowIVb8AgFbCcQDV1NSob9++WrBgQaNjRowYocOHD/un995774qaBAC0Po5vQsjJyVFOTs5Fx7jdbiUnJwfdFACg9WuSa0CFhYVKTExUz549NW3aNB07dqzRsbW1tfJ6vQETAKD1C3kAjRgxQu+88442bNigf/7nf1ZRUZFycnJUX1/f4Pj8/Hx5PB7/lJaWFuqWAADNUMg/BzR+/Hj/n2+++Wb16dNHGRkZKiws1LBhwy4YP3v2bM2aNcv/2uv1EkIAcA1o8tuwu3fvrvj4eO3bt6/B5W63W3FxcQETAKD1a/IAOnjwoI4dO6aUlJSm3hQAoAVx/BbciRMnAs5mysrKtHPnTnXs2FEdO3bUiy++qHHjxik5OVmlpaV68skndf311ys7OzukjQMAWjbHAbR9+3bdcccd/tffX7+ZOHGiFi5cqF27dunf//3fdfz4caWmpmr48OH67W9/K7fbHbquAQAtnuMAGjp0qIwxjS7/9NNPr6ghwKb6r0sc10R87Xw7HQpcjmtGRf3Ucc3+P/VwXCNJn2QudFzTOaJtUNtyKtIV7rjmdMfgrjZ4gqrC5eJZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAi5F/JDeAyXOSJ8o2W1NY6run6y68c10jSnW/NdFzz3/csCmpbuHZxBgQAsIIAAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVvAwUqAVc0VGBVfXtj7EnYTOrjPOe0vcXtMEneBKcQYEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFbwMFKgFSt545ag6v572MLQNhJCMx6f7rgm+vNtTdAJrhRnQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQ8jRasUFhsbXF17T4g7adiRu9Ic19w1/TPHNf+RuMBxzTlX53fTD04kOq6J2/KN45qzjitwNXAGBACwggACAFjhKIDy8/N12223KTY2VomJiRozZoxKSkoCxpw+fVq5ubnq1KmTYmJiNG7cOFVWVoa0aQBAy+cogIqKipSbm6utW7dq3bp1qqur0/Dhw1VTU+MfM3PmTH388cdasWKFioqKdOjQIY0dOzbkjQMAWjZHNyGsXbs24PWSJUuUmJioHTt2aMiQIaqqqtK//du/admyZbrzzjslSYsXL9ZNN92krVu36p/+6Z9C1zkAoEW7omtAVVVVkqSOHTtKknbs2KG6ujplZWX5x9x4443q0qWLtmzZ0uA6amtr5fV6AyYAQOsXdAD5fD7NmDFDAwcOVO/evSVJFRUVioqKUvv27QPGJiUlqaKiosH15Ofny+Px+Ke0NOe3pwIAWp6gAyg3N1e7d+/W8uXLr6iB2bNnq6qqyj+Vl5df0foAAC1DUB9EzcvL0+rVq7Vp0yZ17tzZPz85OVlnzpzR8ePHA86CKisrlZyc3OC63G633G53MG0AAFowR2dAxhjl5eVp5cqV2rhxo9LT0wOW9+vXT5GRkdqwYYN/XklJiQ4cOKABAwaEpmMAQKvg6AwoNzdXy5Yt06pVqxQbG+u/ruPxeNS2bVt5PB796le/0qxZs9SxY0fFxcVp+vTpGjBgAHfAAQACOAqghQsXSpKGDh0aMH/x4sWaNGmSJOn3v/+9wsLCNG7cONXW1io7O1tvvvlmSJoFALQeLmOMsd3E+bxerzwej4ZqtCJckbbbuSaE9b0pqLo9uTGOa5LT/q/jmiMlCY5rJt9Z6LhGkp7q9HVQdQhOn88nOa7p8ouvQt8IQuqsqVOhVqmqqkpxcXGNjuNZcAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALAiqG9ERfPl6tfLcU3b3x8Jalv/nfFuUHWO9bk6m2nuak2d45pIV3hQ26qsr3VcM+dQjuOazq8F1x9aB86AAABWEEAAACsIIACAFQQQAMAKAggAYAUBBACwggACAFhBAAEArCCAAABWEEAAACsIIACAFQQQAMAKHkbaytR0jXFc8373fw1ya1FB1jU9n3xB1c08NNhxzW8S1zuuyf4813FNbGG045rqbo5LJEnps7cEUVXtuCJMO4PYDloLzoAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoeRtrKRH+4zXHNLw4/HNS2jv64neMaXxDPL61z/nxVvf1//uC8SFLpbacd10z78VTHNek7dzmukTGOS+KdbwW4ajgDAgBYQQABAKwggAAAVhBAAAArCCAAgBUEEADACgIIAGAFAQQAsIIAAgBYQQABAKwggAAAVhBAAAArXMYE8YTDJuT1euXxeDRUoxXhirTdDgDAobOmToVapaqqKsXFxTU6jjMgAIAVBBAAwApHAZSfn6/bbrtNsbGxSkxM1JgxY1RSUhIwZujQoXK5XAHTww8H930zAIDWy1EAFRUVKTc3V1u3btW6detUV1en4cOHq6amJmDclClTdPjwYf80b968kDYNAGj5HH0j6tq1awNeL1myRImJidqxY4eGDBninx8dHa3k5OTQdAgAaJWu6BpQVVWVJKljx44B85cuXar4+Hj17t1bs2fP1smTJxtdR21trbxeb8AEAGj9HJ0Bnc/n82nGjBkaOHCgevfu7Z9///33q2vXrkpNTdWuXbv01FNPqaSkRB9++GGD68nPz9eLL74YbBsAgBYq6M8BTZs2TZ988ok2b96szp07Nzpu48aNGjZsmPbt26eMjIwLltfW1qq2ttb/2uv1Ki0tjc8BAUALdbmfAwrqDCgvL0+rV6/Wpk2bLho+kpSZmSlJjQaQ2+2W2+0Opg0AQAvmKICMMZo+fbpWrlypwsJCpaenX7Jm586dkqSUlJSgGgQAtE6OAig3N1fLli3TqlWrFBsbq4qKCkmSx+NR27ZtVVpaqmXLlmnkyJHq1KmTdu3apZkzZ2rIkCHq06dPk/wAAICWydE1IJfL1eD8xYsXa9KkSSovL9eDDz6o3bt3q6amRmlpabr33nv17LPPXvR9wPPxLDgAaNma5BrQpbIqLS1NRUVFTlYJALhG8Sw4AIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAVEbYb+CFjjCTprOokY7kZAIBjZ1Un6X/+P29Mswug6upqSdJmrbHcCQDgSlRXV8vj8TS63GUuFVFXmc/n06FDhxQbGyuXyxWwzOv1Ki0tTeXl5YqLi7PUoX3sh3PYD+ewH85hP5zTHPaDMUbV1dVKTU1VWFjjV3qa3RlQWFiYOnfufNExcXFx1/QB9j32wznsh3PYD+ewH86xvR8udubzPW5CAABYQQABAKxoUQHkdrs1Z84cud1u261YxX44h/1wDvvhHPbDOS1pPzS7mxAAANeGFnUGBABoPQggAIAVBBAAwAoCCABgBQEEALCixQTQggUL1K1bN7Vp00aZmZn64osvbLd01b3wwgtyuVwB04033mi7rSa3adMmjRo1SqmpqXK5XProo48Clhtj9PzzzyslJUVt27ZVVlaW9u7da6fZJnSp/TBp0qQLjo8RI0bYabaJ5Ofn67bbblNsbKwSExM1ZswYlZSUBIw5ffq0cnNz1alTJ8XExGjcuHGqrKy01HHTuJz9MHTo0AuOh4cffthSxw1rEQH0/vvva9asWZozZ46+/PJL9e3bV9nZ2Tpy5Ijt1q66Xr166fDhw/5p8+bNtltqcjU1Nerbt68WLFjQ4PJ58+bp9ddf16JFi7Rt2za1a9dO2dnZOn369FXutGldaj9I0ogRIwKOj/fee+8qdtj0ioqKlJubq61bt2rdunWqq6vT8OHDVVNT4x8zc+ZMffzxx1qxYoWKiop06NAhjR071mLXoXc5+0GSpkyZEnA8zJs3z1LHjTAtQP/+/U1ubq7/dX19vUlNTTX5+fkWu7r65syZY/r27Wu7DaskmZUrV/pf+3w+k5ycbF5++WX/vOPHjxu3223ee+89Cx1eHT/cD8YYM3HiRDN69Ggr/dhy5MgRI8kUFRUZY8793UdGRpoVK1b4x/zjH/8wksyWLVtstdnkfrgfjDHm9ttvN4899pi9pi5Dsz8DOnPmjHbs2KGsrCz/vLCwMGVlZWnLli0WO7Nj7969Sk1NVffu3fXAAw/owIEDtluyqqysTBUVFQHHh8fjUWZm5jV5fBQWFioxMVE9e/bUtGnTdOzYMdstNamqqipJUseOHSVJO3bsUF1dXcDxcOONN6pLly6t+nj44X743tKlSxUfH6/evXtr9uzZOnnypI32GtXsnob9Q99++63q6+uVlJQUMD8pKUl79uyx1JUdmZmZWrJkiXr27KnDhw/rxRdf1ODBg7V7927Fxsbabs+KiooKSWrw+Ph+2bVixIgRGjt2rNLT01VaWqpnnnlGOTk52rJli8LDw223F3I+n08zZszQwIED1bt3b0nnjoeoqCi1b98+YGxrPh4a2g+SdP/996tr165KTU3Vrl279NRTT6mkpEQffvihxW4DNfsAwv/Iycnx/7lPnz7KzMxU165d9cEHH+hXv/qVxc7QHIwfP97/55tvvll9+vRRRkaGCgsLNWzYMIudNY3c3Fzt3r37mrgOejGN7YeHHnrI/+ebb75ZKSkpGjZsmEpLS5WRkXG122xQs38LLj4+XuHh4RfcxVJZWank5GRLXTUP7du3V48ePbRv3z7brVjz/THA8XGh7t27Kz4+vlUeH3l5eVq9erUKCgoCvj8sOTlZZ86c0fHjxwPGt9bjobH90JDMzExJalbHQ7MPoKioKPXr108bNmzwz/P5fNqwYYMGDBhgsTP7Tpw4odLSUqWkpNhuxZr09HQlJycHHB9er1fbtm275o+PgwcP6tixY63q+DDGKC8vTytXrtTGjRuVnp4esLxfv36KjIwMOB5KSkp04MCBVnU8XGo/NGTnzp2S1LyOB9t3QVyO5cuXG7fbbZYsWWL+/ve/m4ceesi0b9/eVFRU2G7tqnr88cdNYWGhKSsrM5999pnJysoy8fHx5siRI7Zba1LV1dWmuLjYFBcXG0nm1VdfNcXFxeabb74xxhjzu9/9zrRv396sWrXK7Nq1y4wePdqkp6ebU6dOWe48tC62H6qrq80TTzxhtmzZYsrKysz69evNT37yE3PDDTeY06dP2249ZKZNm2Y8Ho8pLCw0hw8f9k8nT570j3n44YdNly5dzMaNG8327dvNgAEDzIABAyx2HXqX2g/79u0zL730ktm+fbspKyszq1atMt27dzdDhgyx3HmgFhFAxhjzxhtvmC5dupioqCjTv39/s3XrVtstXXX33XefSUlJMVFRUea6664z9913n9m3b5/ttppcQUGBkXTBNHHiRGPMuVuxn3vuOZOUlGTcbrcZNmyYKSkpsdt0E7jYfjh58qQZPny4SUhIMJGRkaZr165mypQpre6XtIZ+fklm8eLF/jGnTp0yjzzyiOnQoYOJjo429957rzl8+LC9ppvApfbDgQMHzJAhQ0zHjh2N2+02119/vfnNb35jqqqq7Db+A3wfEADAimZ/DQgA0DoRQAAAKwggAIAVBBAAwAoCCABgBQEEALCCAAIAWEEAAQCsIIAAAFYQQAAAKwggAIAV/w/hgVLrpVGHsAAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.figure()\n", + "plt.title(\"Prediction: {}\".format(prediction))\n", + "plt.imshow(img)\n", + "plt.show()" + ] + }, + { + "cell_type": "markdown", + "id": "6377f41a-5654-410b-8bad-d392e9dce7b8", + "metadata": { + "tags": [] + }, + "source": [ + "#### Stop Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": 64, + "id": "9c9fd967-5cd9-4265-add9-db5c1ccf9893", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "[True]" + ] + }, + "execution_count": 64, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def stop_triton(it):\n", + " import docker\n", + " import time\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " print(\">>>> stopping containers: {}\".format([c.short_id for c in containers]))\n", + " if containers:\n", + " container=containers[0]\n", + " container.stop(timeout=120)\n", + "\n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(stop_triton).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 65, + "id": "f612dc0b-538f-4ecf-81f7-ef6b58c493ab", + "metadata": {}, + "outputs": [], + "source": [ + "spark.stop()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "490fc849-e47a-48d7-accc-429ff1cced6b", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "spark-dl-tf", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/keras-metadata.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/keras-metadata_tf.ipynb similarity index 58% rename from examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/keras-metadata.ipynb rename to examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/keras-metadata_tf.ipynb index f9db34ca8..e0683e388 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/keras-metadata.ipynb +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/keras-metadata_tf.ipynb @@ -9,12 +9,36 @@ "From: https://docs.databricks.com/_static/notebooks/deep-learning/keras-metadata.html" ] }, + { + "cell_type": "markdown", + "id": "858e3a8d", + "metadata": {}, + "source": [ + "### Using TensorFlow\n", + "Note that cuFFT/cuDNN/cuBLAS registration errors are expected with `tf=2.17.0` and will not affect behavior, as noted in [this issue.](https://github.com/tensorflow/tensorflow/issues/62075) \n", + "This notebook does not demonstrate inference with TensorRT, as [TF-TRT](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#tensorrt-10) does not yet support `tf=2.17.0`. See the `pytorch` notebooks for TensorRT demos." + ] + }, { "cell_type": "code", - "execution_count": 1, + "execution_count": 2, "id": "cf329ac8-0763-44bc-b0f6-b634b7dc480e", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:41:30.112764: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", + "2024-10-03 17:41:30.119504: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", + "2024-10-03 17:41:30.126948: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", + "2024-10-03 17:41:30.129111: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n", + "2024-10-03 17:41:30.134946: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", + "To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", + "2024-10-03 17:41:30.497048: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" + ] + } + ], "source": [ "import os\n", "import shutil\n", @@ -28,12 +52,61 @@ "import tensorflow as tf\n", "from tensorflow.keras.applications.resnet50 import ResNet50\n", " \n", - "from pyspark.sql.functions import col, pandas_udf, PandasUDFType" + "from pyspark.sql.functions import col, pandas_udf, PandasUDFType\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf" ] }, { "cell_type": "code", - "execution_count": 2, + "execution_count": null, + "id": "44d72768", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "833e36bc", + "metadata": {}, + "outputs": [], + "source": [ + "# Enable GPU memory growth\n", + "gpus = tf.config.experimental.list_physical_devices('GPU')\n", + "if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, "id": "950b0470-a21e-4778-a80e-b8f6ef792dff", "metadata": {}, "outputs": [], @@ -60,10 +133,26 @@ }, { "cell_type": "code", - "execution_count": 3, + "execution_count": 6, "id": "2ddc715a-cdbc-4c49-93e9-58c9d88511da", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:41:32.482802: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2021] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 45311 MB memory: -> device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:01:00.0, compute capability: 8.6\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading data from https://storage.googleapis.com/tensorflow/keras-applications/resnet/resnet50_weights_tf_dim_ordering_tf_kernels.h5\n", + "\u001b[1m102967424/102967424\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 0us/step\n" + ] + } + ], "source": [ "model = ResNet50()\n", "bc_model_weights = sc.broadcast(model.get_weights())" @@ -79,10 +168,19 @@ }, { "cell_type": "code", - "execution_count": 4, + "execution_count": 7, "id": "c0738bec-97d4-4946-8c49-5e6d07ff1afc", "metadata": {}, - "outputs": [], + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading data from https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\n", + "\u001b[1m228813984/228813984\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 0us/step\n" + ] + } + ], "source": [ "import pathlib\n", "dataset_url = \"https://storage.googleapis.com/download.tensorflow.org/example_images/flower_photos.tgz\"\n", @@ -94,7 +192,7 @@ }, { "cell_type": "code", - "execution_count": 5, + "execution_count": 8, "id": "014644f4-2a45-4474-8afb-0daf90043253", "metadata": {}, "outputs": [ @@ -113,7 +211,7 @@ }, { "cell_type": "code", - "execution_count": 6, + "execution_count": 9, "id": "d54f470a-d308-4426-8ed0-33f95155bb4f", "metadata": {}, "outputs": [ @@ -123,7 +221,7 @@ "2048" ] }, - "execution_count": 6, + "execution_count": 9, "metadata": {}, "output_type": "execute_result" } @@ -137,7 +235,7 @@ }, { "cell_type": "code", - "execution_count": 7, + "execution_count": 10, "id": "fd883dc0-4846-4411-a4d6-4f5f252ac707", "metadata": {}, "outputs": [ @@ -145,7 +243,7 @@ "name": "stdout", "output_type": "stream", "text": [ - "/home/leey/.keras/datasets/flower_photos\n" + "/home/rishic/.keras/datasets/flower_photos\n" ] } ], @@ -155,7 +253,7 @@ }, { "cell_type": "code", - "execution_count": 8, + "execution_count": 11, "id": "64f94ee0-f1ea-47f6-a77e-be8da5d1b87a", "metadata": {}, "outputs": [], @@ -184,7 +282,7 @@ }, { "cell_type": "code", - "execution_count": 9, + "execution_count": 12, "id": "670328e3-7274-4d78-b315-487750166a3f", "metadata": {}, "outputs": [ @@ -194,54 +292,14 @@ "0" ] }, - "execution_count": 9, + "execution_count": 12, "metadata": {}, "output_type": "execute_result" } ], "source": [ - "subprocess.call(\"rm -rf resnet50_model\".split())" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "39c663a9-0526-4573-b7b0-27920e5b0004", - "metadata": { - "scrolled": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "WARNING:tensorflow:Compiled the loaded model, but the compiled metrics have yet to be built. `model.compile_metrics` will be empty until you train or evaluate the model.\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING:absl:Found untraced functions such as _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op, _jit_compiled_convolution_op while saving (showing 5 of 53). These functions will not be directly callable after loading.\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "INFO:tensorflow:Assets written to: resnet50_model/assets\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "INFO:tensorflow:Assets written to: resnet50_model/assets\n" - ] - } - ], - "source": [ - "model.save('resnet50_model')" + "subprocess.call(\"rm -rf resnet50_model\".split())\n", + "model.export(\"resnet50_model\", verbose=0)" ] }, { @@ -254,30 +312,16 @@ }, { "cell_type": "code", - "execution_count": 11, + "execution_count": 14, "id": "8ddc22d0-b88a-4906-bd47-bf247e34feeb", "metadata": {}, "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[Stage 1:> (0 + 8) / 8]\r" - ] - }, { "name": "stdout", "output_type": "stream", "text": [ "2048\n" ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] } ], "source": [ @@ -288,7 +332,7 @@ }, { "cell_type": "code", - "execution_count": 12, + "execution_count": 15, "id": "c7adf1d9-1fa7-4456-ae32-cf7d1d43bfd3", "metadata": {}, "outputs": [], @@ -299,7 +343,7 @@ }, { "cell_type": "code", - "execution_count": 13, + "execution_count": 16, "id": "97173c07-a96e-4262-b60f-82865b997e99", "metadata": {}, "outputs": [ @@ -327,7 +371,7 @@ }, { "cell_type": "code", - "execution_count": 14, + "execution_count": 17, "id": "a67b3128-13c1-44f1-a0c0-7cf7a836fee3", "metadata": {}, "outputs": [], @@ -341,7 +385,7 @@ }, { "cell_type": "code", - "execution_count": 15, + "execution_count": 18, "id": "7b33185f-6d1e-4ca9-9757-fdc3d736496b", "metadata": {}, "outputs": [ @@ -349,7 +393,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "/home/leey/devpub/spark/python/pyspark/sql/pandas/functions.py:399: UserWarning: In Python 3.6+ and Spark 3.0+, it is preferred to specify type hints for pandas UDF instead of specifying pandas UDF type which will be deprecated in the future releases. See SPARK-28264 for more details.\n", + "/home/rishic/anaconda3/envs/spark-dl-tf/lib/python3.11/site-packages/pyspark/sql/pandas/functions.py:407: UserWarning: In Python 3.6+ and Spark 3.0+, it is preferred to specify type hints for pandas UDF instead of specifying pandas UDF type which will be deprecated in the future releases. See SPARK-28264 for more details.\n", " warnings.warn(\n" ] } @@ -357,6 +401,16 @@ "source": [ "@pandas_udf(ArrayType(FloatType()), PandasUDFType.SCALAR_ITER)\n", "def predict_batch_udf(image_batch_iter):\n", + "\n", + " # Enable GPU memory growth to avoid CUDA OOM\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + "\n", " batch_size = 64\n", " model = ResNet50(weights=None)\n", " model.set_weights(bc_model_weights.value)\n", @@ -371,7 +425,7 @@ }, { "cell_type": "code", - "execution_count": 16, + "execution_count": 19, "id": "ad8c05da-db38-45ef-81d0-1f862f575ced", "metadata": {}, "outputs": [ @@ -379,7 +433,7 @@ "name": "stderr", "output_type": "stream", "text": [ - "[Stage 7:============================================> (3 + 1) / 4]\r" + " \r" ] }, { @@ -389,38 +443,31 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| prediction|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "|[1.7195931E-4, 3.2735552E-4, 7.533328E-5, 1.9810574E-4, 6.6188135E-5, 4.304618E-4, 1.3097588E-5, 5.2791635E-5, 2.3571...|\n", - "|[7.738322E-5, 5.935413E-4, 1.3075814E-4, 1.8000253E-4, 1.3001164E-4, 2.2790757E-4, 5.0780895E-5, 2.3768713E-4, 3.8676...|\n", - "|[2.3994662E-5, 3.4409956E-4, 1.8440963E-4, 1.6311614E-4, 1.4301096E-4, 8.883273E-5, 2.6071444E-5, 2.3609246E-4, 1.481...|\n", - "|[7.33013E-5, 2.6747186E-4, 1.2007599E-4, 2.0717822E-4, 8.630799E-5, 2.2318888E-4, 1.3459768E-5, 8.564859E-5, 1.482114...|\n", - "|[5.592278E-5, 2.5165555E-4, 1.3106635E-4, 1.5746983E-4, 8.4952095E-5, 2.1269226E-4, 9.83865E-6, 1.6246884E-4, 1.24666...|\n", - "|[1.1709497E-4, 3.0075156E-4, 9.33239E-5, 1.6481965E-4, 6.906736E-5, 4.821755E-4, 8.922642E-6, 5.9618797E-5, 1.72143E-...|\n", - "|[1.1801151E-4, 2.1426812E-4, 5.9151847E-5, 1.1427055E-4, 5.1385836E-5, 4.3937864E-4, 4.607807E-6, 3.3783803E-5, 8.869...|\n", - "|[1.04612736E-4, 2.7663654E-4, 9.435623E-5, 1.6525248E-4, 7.4158685E-5, 4.257633E-4, 8.704045E-6, 5.3063635E-5, 1.2789...|\n", - "|[7.352998E-5, 3.3315457E-4, 7.0680355E-5, 1.7582536E-4, 7.606573E-5, 2.8071608E-4, 1.5708138E-5, 7.895344E-5, 1.63833...|\n", - "|[5.4112577E-5, 3.4097856E-4, 4.273796E-5, 6.867077E-5, 4.2725427E-5, 3.6373304E-4, 4.102062E-6, 1.15471645E-4, 1.3269...|\n", - "|[1.6299986E-5, 3.011269E-4, 1.0530894E-4, 1.0999596E-4, 6.247357E-5, 8.1669314E-5, 8.867516E-6, 1.9714379E-4, 9.89142...|\n", - "|[2.4569267E-4, 5.179993E-4, 1.7478937E-4, 2.2669601E-4, 1.2160459E-4, 4.5433E-4, 3.953984E-5, 7.979992E-5, 3.3513832E...|\n", - "|[9.029223E-5, 3.9454165E-4, 1.665605E-4, 1.9196334E-4, 1.1372031E-4, 2.0514896E-4, 2.7878115E-5, 1.4796885E-4, 3.3018...|\n", - "|[8.730619E-6, 3.4874916E-4, 5.797578E-5, 4.9443646E-5, 5.4349573E-5, 4.1568164E-5, 1.4966195E-5, 6.196577E-4, 2.33120...|\n", - "|[2.1702444E-5, 2.3885888E-4, 1.0117506E-4, 7.988898E-5, 7.155834E-5, 1.2449219E-4, 7.99208E-6, 2.611562E-4, 1.5751371...|\n", - "|[8.29938E-5, 3.1705064E-4, 1.587315E-4, 2.3244774E-4, 9.55335E-5, 1.9356904E-4, 1.7146796E-5, 9.8108874E-5, 1.8489505...|\n", - "|[6.3181215E-5, 2.6232892E-4, 1.0581372E-4, 2.0251807E-4, 6.83598E-5, 2.0704906E-4, 1.1209849E-5, 8.139733E-5, 1.70928...|\n", - "|[1.5359989E-4, 3.3213175E-4, 7.3188654E-5, 1.3153376E-4, 6.174973E-5, 3.554963E-4, 1.0753482E-5, 6.729857E-5, 2.27700...|\n", - "|[7.3694064E-5, 3.3086358E-4, 1.6089129E-4, 1.9742687E-4, 1.0419457E-4, 2.0769038E-4, 1.9587573E-5, 1.2341993E-4, 1.45...|\n", - "|[6.0036982E-5, 3.690938E-4, 9.357083E-5, 1.2322696E-4, 7.118597E-5, 2.5646162E-4, 1.1585914E-5, 2.2628417E-4, 2.26159...|\n", + "|[1.2938889E-4, 2.4666305E-4, 6.765791E-5, 1.2263245E-4, 5.7486624E-5, 3.9616702E-4, 7.0566134E-6, 4.1722178E-5, 1.225...|\n", + "|[4.4501914E-5, 3.5403698E-4, 4.6702033E-5, 8.102543E-5, 3.1704556E-5, 1.9194305E-4, 7.905952E-6, 1.3744082E-4, 1.9563...|\n", + "|[1.05672516E-4, 2.2686279E-4, 3.0055395E-5, 6.523785E-5, 2.352077E-5, 3.7122983E-4, 3.3315896E-6, 2.2584054E-5, 9.775...|\n", + "|[2.0331638E-5, 2.2746396E-4, 7.828012E-5, 6.986782E-5, 4.705316E-5, 9.80732E-5, 5.561918E-6, 2.3519044E-4, 1.3803913E...|\n", + "|[1.130241E-4, 2.3187004E-4, 5.296914E-5, 1.0871329E-4, 4.027478E-5, 3.7183522E-4, 5.5931855E-6, 3.4792112E-5, 1.14155...|\n", + "|[9.094467E-5, 2.06384E-4, 4.514821E-5, 7.665891E-5, 3.2262324E-5, 3.3875552E-4, 3.831814E-6, 4.1848412E-5, 9.94389E-6...|\n", + "|[1.07847634E-4, 3.7848807E-4, 7.660533E-5, 1.2446754E-4, 4.7595917E-5, 3.333814E-4, 1.0669675E-5, 9.133265E-5, 1.8015...|\n", + "|[2.2261223E-5, 2.734666E-4, 3.8122747E-5, 6.2266954E-5, 1.7935155E-5, 1.7268128E-4, 6.034271E-6, 1.06450585E-4, 1.789...|\n", + "|[1.1065645E-4, 2.900581E-4, 4.2585547E-5, 1.074203E-4, 3.052314E-5, 4.794604E-4, 6.4872897E-6, 3.646897E-5, 1.3717402...|\n", + "|[9.673917E-5, 2.058331E-4, 7.4652424E-5, 1.1323769E-4, 4.6106186E-5, 2.8604185E-4, 5.62365E-6, 5.471466E-5, 9.664386E...|\n", + "|[7.411196E-5, 3.291524E-4, 1.3454164E-4, 1.7738447E-4, 8.467504E-5, 2.2466244E-4, 1.3621126E-5, 1.1778668E-4, 1.83372...|\n", + "|[8.721524E-5, 2.7338538E-4, 3.5964815E-5, 7.792533E-5, 2.3559302E-5, 3.6789547E-4, 3.5665628E-6, 3.648153E-5, 1.07589...|\n", + "|[9.723709E-5, 2.7619812E-4, 5.7464153E-5, 1.10104906E-4, 3.8317143E-5, 3.490506E-4, 6.1553183E-6, 4.413095E-5, 1.1236...|\n", + "|[6.940235E-5, 2.5377885E-4, 5.057188E-5, 1.1485363E-4, 3.0059196E-5, 2.7862669E-4, 5.024019E-6, 5.1511077E-5, 1.16149...|\n", + "|[4.2095784E-5, 2.4891715E-4, 1.236292E-4, 1.4306813E-4, 7.3354306E-5, 1.6047148E-4, 7.958807E-6, 1.3556339E-4, 1.4698...|\n", + "|[2.7327887E-5, 3.8553146E-4, 1.2939748E-4, 1.5762268E-4, 7.307493E-5, 8.5530424E-5, 1.2648808E-5, 1.9154618E-4, 2.307...|\n", + "|[3.036101E-5, 3.5572305E-4, 1.600718E-4, 2.1437313E-4, 8.063033E-5, 1.02061334E-4, 1.3876456E-5, 1.561292E-4, 1.63637...|\n", + "|[3.3109587E-5, 2.8182982E-4, 1.7998899E-4, 2.0246049E-4, 1.3720036E-4, 1.01000114E-4, 3.427488E-5, 3.887249E-4, 3.189...|\n", + "|[4.549448E-5, 2.8782588E-4, 2.3703449E-4, 2.448979E-4, 1.20997625E-4, 1.3744453E-4, 1.62803E-5, 2.2094708E-4, 1.56962...|\n", + "|[1.2242574E-4, 2.8095162E-4, 6.332559E-5, 1.0209269E-4, 4.335324E-5, 3.906304E-4, 8.205706E-6, 6.202823E-5, 1.5312888...|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n", - "CPU times: user 42.3 ms, sys: 2.17 ms, total: 44.5 ms\n", - "Wall time: 12.3 s\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" + "CPU times: user 10.8 ms, sys: 4.93 ms, total: 15.7 ms\n", + "Wall time: 9.25 s\n" ] } ], @@ -432,7 +479,7 @@ }, { "cell_type": "code", - "execution_count": 17, + "execution_count": 20, "id": "40799f8e-443e-40ca-919b-391f901cb3f4", "metadata": {}, "outputs": [ @@ -447,8 +494,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 26 ms, sys: 8.71 ms, total: 34.7 ms\n", - "Wall time: 49.9 s\n" + "CPU times: user 9.96 ms, sys: 3.32 ms, total: 13.3 ms\n", + "Wall time: 14 s\n" ] }, { @@ -476,7 +523,7 @@ }, { "cell_type": "code", - "execution_count": 18, + "execution_count": 21, "id": "e6af27b2-ddc0-42ee-94cc-9ba5ffee6868", "metadata": {}, "outputs": [], @@ -488,7 +535,7 @@ }, { "cell_type": "code", - "execution_count": 19, + "execution_count": 22, "id": "dda88b46-6300-4bf7-bc10-7403f4fbbf92", "metadata": {}, "outputs": [], @@ -496,6 +543,16 @@ "def predict_batch_fn():\n", " import tensorflow as tf\n", " from tensorflow.keras.applications.resnet50 import ResNet50\n", + "\n", + " # Enable GPU memory growth\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + "\n", " model = ResNet50()\n", " def predict(inputs):\n", " inputs = inputs * (2. / 255) - 1\n", @@ -505,7 +562,7 @@ }, { "cell_type": "code", - "execution_count": 20, + "execution_count": 23, "id": "cff0e851-563d-40b6-9d05-509c22b3b7f9", "metadata": {}, "outputs": [], @@ -518,7 +575,7 @@ }, { "cell_type": "code", - "execution_count": 21, + "execution_count": 24, "id": "f733c38b-867d-48c1-b9a6-74a931561896", "metadata": {}, "outputs": [], @@ -529,7 +586,7 @@ }, { "cell_type": "code", - "execution_count": 22, + "execution_count": 25, "id": "aa7c156f-e2b3-4837-9427-ccf3a5720412", "metadata": {}, "outputs": [], @@ -539,7 +596,7 @@ }, { "cell_type": "code", - "execution_count": 23, + "execution_count": 26, "id": "80bc50ad-eaf5-4fce-a354-5e17d65e2da5", "metadata": { "tags": [] @@ -559,31 +616,31 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| prediction|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "|[1.7195931E-4, 3.2735552E-4, 7.533328E-5, 1.9810574E-4, 6.6188135E-5, 4.304618E-4, 1.3097588E-5, 5.2791635E-5, 2.3571...|\n", - "|[7.738322E-5, 5.935413E-4, 1.3075814E-4, 1.8000253E-4, 1.3001164E-4, 2.2790757E-4, 5.0780895E-5, 2.3768713E-4, 3.8676...|\n", - "|[2.3994662E-5, 3.4409956E-4, 1.8440963E-4, 1.6311614E-4, 1.4301096E-4, 8.883273E-5, 2.6071444E-5, 2.3609246E-4, 1.481...|\n", - "|[7.33013E-5, 2.6747186E-4, 1.2007599E-4, 2.0717822E-4, 8.630799E-5, 2.2318888E-4, 1.3459768E-5, 8.564859E-5, 1.482114...|\n", - "|[5.592278E-5, 2.5165555E-4, 1.3106635E-4, 1.5746983E-4, 8.4952095E-5, 2.1269226E-4, 9.83865E-6, 1.6246884E-4, 1.24666...|\n", - "|[1.1709497E-4, 3.0075156E-4, 9.33239E-5, 1.6481965E-4, 6.906736E-5, 4.821755E-4, 8.922642E-6, 5.9618797E-5, 1.72143E-...|\n", - "|[1.1801151E-4, 2.1426812E-4, 5.9151847E-5, 1.1427055E-4, 5.1385836E-5, 4.3937864E-4, 4.607807E-6, 3.3783803E-5, 8.869...|\n", - "|[1.04612736E-4, 2.7663654E-4, 9.435623E-5, 1.6525248E-4, 7.4158685E-5, 4.257633E-4, 8.704045E-6, 5.3063635E-5, 1.2789...|\n", - "|[7.352998E-5, 3.3315457E-4, 7.0680355E-5, 1.7582536E-4, 7.606573E-5, 2.8071608E-4, 1.5708138E-5, 7.895344E-5, 1.63833...|\n", - "|[5.4112577E-5, 3.4097856E-4, 4.273796E-5, 6.867077E-5, 4.2725427E-5, 3.6373304E-4, 4.102062E-6, 1.15471645E-4, 1.3269...|\n", - "|[1.6299986E-5, 3.011269E-4, 1.0530894E-4, 1.0999596E-4, 6.247357E-5, 8.1669314E-5, 8.867516E-6, 1.9714379E-4, 9.89142...|\n", - "|[2.4569267E-4, 5.179993E-4, 1.7478937E-4, 2.2669601E-4, 1.2160459E-4, 4.5433E-4, 3.953984E-5, 7.979992E-5, 3.3513832E...|\n", - "|[9.029223E-5, 3.9454165E-4, 1.665605E-4, 1.9196334E-4, 1.1372031E-4, 2.0514896E-4, 2.7878115E-5, 1.4796885E-4, 3.3018...|\n", - "|[8.730619E-6, 3.4874916E-4, 5.797578E-5, 4.9443646E-5, 5.4349573E-5, 4.1568164E-5, 1.4966195E-5, 6.196577E-4, 2.33120...|\n", - "|[2.1702444E-5, 2.3885888E-4, 1.0117506E-4, 7.988898E-5, 7.155834E-5, 1.2449219E-4, 7.99208E-6, 2.611562E-4, 1.5751371...|\n", - "|[8.29938E-5, 3.1705064E-4, 1.587315E-4, 2.3244774E-4, 9.55335E-5, 1.9356904E-4, 1.7146796E-5, 9.8108874E-5, 1.8489505...|\n", - "|[6.3181215E-5, 2.6232892E-4, 1.0581372E-4, 2.0251807E-4, 6.83598E-5, 2.0704906E-4, 1.1209849E-5, 8.139733E-5, 1.70928...|\n", - "|[1.5359989E-4, 3.3213175E-4, 7.3188654E-5, 1.3153376E-4, 6.174973E-5, 3.554963E-4, 1.0753482E-5, 6.729857E-5, 2.27700...|\n", - "|[7.3694064E-5, 3.3086358E-4, 1.6089129E-4, 1.9742687E-4, 1.0419457E-4, 2.0769038E-4, 1.9587573E-5, 1.2341993E-4, 1.45...|\n", - "|[6.0036982E-5, 3.690938E-4, 9.357083E-5, 1.2322696E-4, 7.118597E-5, 2.5646162E-4, 1.1585914E-5, 2.2628417E-4, 2.26159...|\n", + "|[1.296447E-4, 2.465122E-4, 6.7463385E-5, 1.2231144E-4, 5.731739E-5, 3.9644213E-4, 7.0297688E-6, 4.1668914E-5, 1.22212...|\n", + "|[4.4481887E-5, 3.526653E-4, 4.683818E-5, 8.1168495E-5, 3.178377E-5, 1.9188467E-4, 7.885617E-6, 1.3758946E-4, 1.956621...|\n", + "|[1.05946536E-4, 2.2744355E-4, 3.0219735E-5, 6.548672E-5, 2.3649674E-5, 3.7177472E-4, 3.353236E-6, 2.271976E-5, 9.8115...|\n", + "|[2.0392703E-5, 2.2817637E-4, 7.840744E-5, 6.9875685E-5, 4.702542E-5, 9.8244425E-5, 5.5829764E-6, 2.3530141E-4, 1.3836...|\n", + "|[1.1312391E-4, 2.31244E-4, 5.279228E-5, 1.0859927E-4, 4.0202678E-5, 3.721753E-4, 5.563934E-6, 3.4674114E-5, 1.1389492...|\n", + "|[9.126345E-5, 2.0679034E-4, 4.5165678E-5, 7.679106E-5, 3.234611E-5, 3.3994843E-4, 3.84E-6, 4.1930372E-5, 9.949454E-6,...|\n", + "|[1.07930486E-4, 3.7741542E-4, 7.613175E-5, 1.2414041E-4, 4.7409427E-5, 3.332554E-4, 1.05853915E-5, 9.1026224E-5, 1.79...|\n", + "|[2.2216762E-5, 2.7354853E-4, 3.8192928E-5, 6.2340725E-5, 1.7952003E-5, 1.7253387E-4, 6.020507E-6, 1.0669143E-4, 1.786...|\n", + "|[1.10480236E-4, 2.89734E-4, 4.239379E-5, 1.0727814E-4, 3.047985E-5, 4.7992737E-4, 6.4530495E-6, 3.6428817E-5, 1.36967...|\n", + "|[9.6864875E-5, 2.0573521E-4, 7.4498465E-5, 1.1323085E-4, 4.6088306E-5, 2.8680824E-4, 5.604823E-6, 5.461046E-5, 9.6629...|\n", + "|[7.4198484E-5, 3.2886668E-4, 1.3441108E-4, 1.7755068E-4, 8.469927E-5, 2.2534095E-4, 1.3617541E-5, 1.1781904E-4, 1.833...|\n", + "|[8.7561886E-5, 2.7312653E-4, 3.5959012E-5, 7.7946424E-5, 2.3565723E-5, 3.6881721E-4, 3.5630535E-6, 3.642736E-5, 1.074...|\n", + "|[9.743975E-5, 2.7615853E-4, 5.74148E-5, 1.10329434E-4, 3.83045E-5, 3.500394E-4, 6.167429E-6, 4.4207005E-5, 1.1250093E...|\n", + "|[6.9320704E-5, 2.53287E-4, 5.0612853E-5, 1.14936556E-4, 3.0210098E-5, 2.7870742E-4, 5.031114E-6, 5.169024E-5, 1.16021...|\n", + "|[4.2203726E-5, 2.4911022E-4, 1.2378568E-4, 1.4274308E-4, 7.32259E-5, 1.6058519E-4, 7.9425035E-6, 1.3519496E-4, 1.4662...|\n", + "|[2.7190901E-5, 3.8381666E-4, 1.2918573E-4, 1.570463E-4, 7.310112E-5, 8.554618E-5, 1.2614603E-5, 1.9213595E-4, 2.30354...|\n", + "|[3.0573912E-5, 3.5561546E-4, 1.5945674E-4, 2.1361349E-4, 8.046549E-5, 1.0269262E-4, 1.3862439E-5, 1.5622783E-4, 1.638...|\n", + "|[3.3117096E-5, 2.8073433E-4, 1.7961214E-4, 2.020287E-4, 1.3662946E-4, 1.0117796E-4, 3.4090703E-5, 3.8897162E-4, 3.181...|\n", + "|[4.5728237E-5, 2.8880237E-4, 2.3783019E-4, 2.4589908E-4, 1.2160292E-4, 1.3812551E-4, 1.6343482E-5, 2.2073709E-4, 1.57...|\n", + "|[1.2280059E-4, 2.806991E-4, 6.3642765E-5, 1.02471764E-4, 4.351664E-5, 3.9150563E-4, 8.235125E-6, 6.211928E-5, 1.53269...|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n", - "CPU times: user 19.7 ms, sys: 7.84 ms, total: 27.5 ms\n", - "Wall time: 7.41 s\n" + "CPU times: user 8.12 ms, sys: 3.38 ms, total: 11.5 ms\n", + "Wall time: 5.59 s\n" ] }, { @@ -603,7 +660,7 @@ }, { "cell_type": "code", - "execution_count": 24, + "execution_count": 27, "id": "41cace80-7a4b-4929-8e63-9c83f9745e02", "metadata": {}, "outputs": [ @@ -621,31 +678,31 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| prediction|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "|[1.7195931E-4, 3.2735552E-4, 7.533328E-5, 1.9810574E-4, 6.6188135E-5, 4.304618E-4, 1.3097588E-5, 5.2791635E-5, 2.3571...|\n", - "|[7.738322E-5, 5.935413E-4, 1.3075814E-4, 1.8000253E-4, 1.3001164E-4, 2.2790757E-4, 5.0780895E-5, 2.3768713E-4, 3.8676...|\n", - "|[2.3994662E-5, 3.4409956E-4, 1.8440963E-4, 1.6311614E-4, 1.4301096E-4, 8.883273E-5, 2.6071444E-5, 2.3609246E-4, 1.481...|\n", - "|[7.33013E-5, 2.6747186E-4, 1.2007599E-4, 2.0717822E-4, 8.630799E-5, 2.2318888E-4, 1.3459768E-5, 8.564859E-5, 1.482114...|\n", - "|[5.592278E-5, 2.5165555E-4, 1.3106635E-4, 1.5746983E-4, 8.4952095E-5, 2.1269226E-4, 9.83865E-6, 1.6246884E-4, 1.24666...|\n", - "|[1.1709497E-4, 3.0075156E-4, 9.33239E-5, 1.6481965E-4, 6.906736E-5, 4.821755E-4, 8.922642E-6, 5.9618797E-5, 1.72143E-...|\n", - "|[1.1801151E-4, 2.1426812E-4, 5.9151847E-5, 1.1427055E-4, 5.1385836E-5, 4.3937864E-4, 4.607807E-6, 3.3783803E-5, 8.869...|\n", - "|[1.04612736E-4, 2.7663654E-4, 9.435623E-5, 1.6525248E-4, 7.4158685E-5, 4.257633E-4, 8.704045E-6, 5.3063635E-5, 1.2789...|\n", - "|[7.352998E-5, 3.3315457E-4, 7.0680355E-5, 1.7582536E-4, 7.606573E-5, 2.8071608E-4, 1.5708138E-5, 7.895344E-5, 1.63833...|\n", - "|[5.4112577E-5, 3.4097856E-4, 4.273796E-5, 6.867077E-5, 4.2725427E-5, 3.6373304E-4, 4.102062E-6, 1.15471645E-4, 1.3269...|\n", - "|[1.6299986E-5, 3.011269E-4, 1.0530894E-4, 1.0999596E-4, 6.247357E-5, 8.1669314E-5, 8.867516E-6, 1.9714379E-4, 9.89142...|\n", - "|[2.4569267E-4, 5.179993E-4, 1.7478937E-4, 2.2669601E-4, 1.2160459E-4, 4.5433E-4, 3.953984E-5, 7.979992E-5, 3.3513832E...|\n", - "|[9.029223E-5, 3.9454165E-4, 1.665605E-4, 1.9196334E-4, 1.1372031E-4, 2.0514896E-4, 2.7878115E-5, 1.4796885E-4, 3.3018...|\n", - "|[8.730619E-6, 3.4874916E-4, 5.797578E-5, 4.9443646E-5, 5.4349573E-5, 4.1568164E-5, 1.4966195E-5, 6.196577E-4, 2.33120...|\n", - "|[2.1702444E-5, 2.3885888E-4, 1.0117506E-4, 7.988898E-5, 7.155834E-5, 1.2449219E-4, 7.99208E-6, 2.611562E-4, 1.5751371...|\n", - "|[8.29938E-5, 3.1705064E-4, 1.587315E-4, 2.3244774E-4, 9.55335E-5, 1.9356904E-4, 1.7146796E-5, 9.8108874E-5, 1.8489505...|\n", - "|[6.3181215E-5, 2.6232892E-4, 1.0581372E-4, 2.0251807E-4, 6.83598E-5, 2.0704906E-4, 1.1209849E-5, 8.139733E-5, 1.70928...|\n", - "|[1.5359989E-4, 3.3213175E-4, 7.3188654E-5, 1.3153376E-4, 6.174973E-5, 3.554963E-4, 1.0753482E-5, 6.729857E-5, 2.27700...|\n", - "|[7.3694064E-5, 3.3086358E-4, 1.6089129E-4, 1.9742687E-4, 1.0419457E-4, 2.0769038E-4, 1.9587573E-5, 1.2341993E-4, 1.45...|\n", - "|[6.0036982E-5, 3.690938E-4, 9.357083E-5, 1.2322696E-4, 7.118597E-5, 2.5646162E-4, 1.1585914E-5, 2.2628417E-4, 2.26159...|\n", + "|[1.293178E-4, 2.4644283E-4, 6.760039E-5, 1.2260793E-4, 5.7431564E-5, 3.9597694E-4, 7.0522524E-6, 4.1717416E-5, 1.2240...|\n", + "|[4.4487308E-5, 3.5378174E-4, 4.6667028E-5, 8.102564E-5, 3.168566E-5, 1.9189132E-4, 7.903805E-6, 1.3741471E-4, 1.95482...|\n", + "|[1.0566196E-4, 2.2684377E-4, 3.00564E-5, 6.5251304E-5, 2.3520754E-5, 3.7116173E-4, 3.331476E-6, 2.2584616E-5, 9.77515...|\n", + "|[2.0337258E-5, 2.2749524E-4, 7.8351426E-5, 6.991163E-5, 4.7081656E-5, 9.8092445E-5, 5.564894E-6, 2.3517481E-4, 1.3805...|\n", + "|[1.12979564E-4, 2.3172122E-4, 5.2946547E-5, 1.0876398E-4, 4.0259067E-5, 3.7143996E-4, 5.5940513E-6, 3.4814777E-5, 1.1...|\n", + "|[9.093228E-5, 2.0639994E-4, 4.5151268E-5, 7.666316E-5, 3.2264295E-5, 3.387436E-4, 3.832487E-6, 4.185193E-5, 9.944773E...|\n", + "|[1.0783461E-4, 3.7850672E-4, 7.660902E-5, 1.2446321E-4, 4.7591406E-5, 3.3328883E-4, 1.067249E-5, 9.131178E-5, 1.80121...|\n", + "|[2.2258617E-5, 2.7345872E-4, 3.814439E-5, 6.229726E-5, 1.79387E-5, 1.7259057E-4, 6.0371217E-6, 1.0649798E-4, 1.789726...|\n", + "|[1.1067773E-4, 2.8997674E-4, 4.2570035E-5, 1.0747747E-4, 3.0524247E-5, 4.7921995E-4, 6.489833E-6, 3.6502548E-5, 1.371...|\n", + "|[9.676251E-5, 2.0588847E-4, 7.467098E-5, 1.1326933E-4, 4.6123736E-5, 2.8609246E-4, 5.627118E-6, 5.4726373E-5, 9.66839...|\n", + "|[7.4104944E-5, 3.290917E-4, 1.3448784E-4, 1.7742367E-4, 8.463227E-5, 2.2462371E-4, 1.3614881E-5, 1.17794625E-4, 1.833...|\n", + "|[8.7211796E-5, 2.7337394E-4, 3.5953894E-5, 7.7924225E-5, 2.3554327E-5, 3.67775E-4, 3.5652213E-6, 3.647724E-5, 1.07577...|\n", + "|[9.7237185E-5, 2.762026E-4, 5.7450008E-5, 1.1019135E-4, 3.831896E-5, 3.4878452E-4, 6.1574788E-6, 4.415526E-5, 1.12374...|\n", + "|[6.938849E-5, 2.5376282E-4, 5.0565883E-5, 1.14880335E-4, 3.0061366E-5, 2.7866007E-4, 5.024482E-6, 5.152425E-5, 1.1617...|\n", + "|[4.2096388E-5, 2.4889092E-4, 1.2363133E-4, 1.4304162E-4, 7.337785E-5, 1.6042824E-4, 7.959722E-6, 1.3552785E-4, 1.4693...|\n", + "|[2.730248E-5, 3.851789E-4, 1.293143E-4, 1.5753493E-4, 7.302161E-5, 8.547956E-5, 1.26348905E-5, 1.9148648E-4, 2.304900...|\n", + "|[3.0354899E-5, 3.5562844E-4, 1.6008675E-4, 2.1440513E-4, 8.062159E-5, 1.02023136E-4, 1.3876455E-5, 1.5611007E-4, 1.63...|\n", + "|[3.3083066E-5, 2.8158593E-4, 1.7979987E-4, 2.0232225E-4, 1.3704685E-4, 1.0091762E-4, 3.4243407E-5, 3.8870922E-4, 3.18...|\n", + "|[4.5485373E-5, 2.878148E-4, 2.3707838E-4, 2.4493985E-4, 1.21028905E-4, 1.3738636E-4, 1.6280053E-5, 2.2104722E-4, 1.56...|\n", + "|[1.22468E-4, 2.809503E-4, 6.3342835E-5, 1.021957E-4, 4.3373006E-5, 3.905496E-4, 8.212427E-6, 6.2081075E-5, 1.5323925E...|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n", - "CPU times: user 31.5 ms, sys: 6.18 ms, total: 37.6 ms\n", - "Wall time: 6.06 s\n" + "CPU times: user 2.75 ms, sys: 3.03 ms, total: 5.78 ms\n", + "Wall time: 4.79 s\n" ] }, { @@ -664,7 +721,7 @@ }, { "cell_type": "code", - "execution_count": 25, + "execution_count": 28, "id": "56a2ec8a-de09-4d7c-9666-1b3c76f10657", "metadata": {}, "outputs": [ @@ -679,8 +736,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 30.8 ms, sys: 4.88 ms, total: 35.7 ms\n", - "Wall time: 49.1 s\n" + "CPU times: user 10.7 ms, sys: 4.25 ms, total: 14.9 ms\n", + "Wall time: 16.9 s\n" ] }, { @@ -709,7 +766,7 @@ }, { "cell_type": "code", - "execution_count": 26, + "execution_count": 29, "id": "2605d134-ef75-4d94-9b16-2c6d85f29bef", "metadata": {}, "outputs": [], @@ -722,7 +779,7 @@ }, { "cell_type": "code", - "execution_count": 27, + "execution_count": 30, "id": "4666e618-8038-4dc5-9be7-793aedbf4500", "metadata": {}, "outputs": [], @@ -747,7 +804,7 @@ }, { "cell_type": "code", - "execution_count": 28, + "execution_count": 31, "id": "8c8c0744-0558-4dac-bbfe-8bdde4b2af2d", "metadata": {}, "outputs": [ @@ -764,7 +821,7 @@ "[True]" ] }, - "execution_count": 28, + "execution_count": 31, "metadata": {}, "output_type": "execute_result" } @@ -785,7 +842,7 @@ " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", " else:\n", " container=client.containers.run(\n", - " \"nvcr.io/nvidia/tritonserver:23.04-py3\", \"tritonserver --model-repository=/models\",\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", " detach=True,\n", " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", " name=\"spark-triton\",\n", @@ -821,7 +878,7 @@ }, { "cell_type": "code", - "execution_count": 29, + "execution_count": 32, "id": "bcd46360-6851-4a9d-8590-c086e001242a", "metadata": {}, "outputs": [], @@ -831,7 +888,7 @@ " import tritonclient.grpc as grpcclient\n", " \n", " np_types = {\n", - " \"BOOL\": np.dtype(np.bool8),\n", + " \"BOOL\": np.dtype(np.bool_),\n", " \"INT8\": np.dtype(np.int8),\n", " \"INT16\": np.dtype(np.int16),\n", " \"INT32\": np.dtype(np.int32),\n", @@ -872,7 +929,7 @@ }, { "cell_type": "code", - "execution_count": 30, + "execution_count": 33, "id": "9fabcaeb-5a44-42bb-8097-5dbc2d0cee3e", "metadata": {}, "outputs": [], @@ -887,7 +944,7 @@ }, { "cell_type": "code", - "execution_count": 31, + "execution_count": 34, "id": "b17f33c8-a0f0-4bce-91f8-5838ba9b12a7", "metadata": {}, "outputs": [], @@ -898,7 +955,7 @@ }, { "cell_type": "code", - "execution_count": 32, + "execution_count": 35, "id": "8e5b9e99-a1cf-43d3-a795-c7271a917057", "metadata": {}, "outputs": [], @@ -908,7 +965,7 @@ }, { "cell_type": "code", - "execution_count": 33, + "execution_count": 36, "id": "e595473d-1a5d-46a6-a6ba-89d2ea903de9", "metadata": { "tags": [] @@ -928,31 +985,31 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| prediction|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "|[1.7042673E-4, 3.267522E-4, 7.581915E-5, 1.9806583E-4, 6.629557E-5, 4.2843417E-4, 1.31030665E-5, 5.2828378E-5, 2.3425...|\n", - "|[7.658893E-5, 5.898446E-4, 1.3134143E-4, 1.7956285E-4, 1.2959713E-4, 2.2623697E-4, 5.0602717E-5, 2.387755E-4, 3.85566...|\n", - "|[2.3643051E-5, 3.3990925E-4, 1.8499317E-4, 1.6320233E-4, 1.4257104E-4, 8.804509E-5, 2.6084534E-5, 2.3692146E-4, 1.479...|\n", - "|[7.265595E-5, 2.660495E-4, 1.2043192E-4, 2.0687138E-4, 8.6514396E-5, 2.2079627E-4, 1.3474356E-5, 8.564823E-5, 1.47982...|\n", - "|[5.583156E-5, 2.5246604E-4, 1.3152408E-4, 1.5752943E-4, 8.486259E-5, 2.1114835E-4, 9.902404E-6, 1.6352077E-4, 1.25058...|\n", - "|[1.1620977E-4, 2.9887763E-4, 9.283447E-5, 1.6428927E-4, 6.8689915E-5, 4.8077543E-4, 8.824636E-6, 5.926726E-5, 1.70530...|\n", - "|[1.1755548E-4, 2.1347901E-4, 5.9497E-5, 1.1462011E-4, 5.154583E-5, 4.378169E-4, 4.606332E-6, 3.3860095E-5, 8.858123E-...|\n", - "|[1.0416699E-4, 2.7650507E-4, 9.4154115E-5, 1.6408198E-4, 7.396577E-5, 4.2134034E-4, 8.686047E-6, 5.308059E-5, 1.27552...|\n", - "|[7.3212854E-5, 3.3290745E-4, 7.060745E-5, 1.7570716E-4, 7.5938726E-5, 2.7863326E-4, 1.5730891E-5, 7.8610516E-5, 1.641...|\n", - "|[5.3438263E-5, 3.4120653E-4, 4.2561773E-5, 6.7946996E-5, 4.2590156E-5, 3.6005717E-4, 4.079908E-6, 1.1698964E-4, 1.331...|\n", - "|[1.6286886E-5, 3.0098078E-4, 1.0513259E-4, 1.0939782E-4, 6.222565E-5, 8.1393446E-5, 8.843026E-6, 1.9695786E-4, 9.9110...|\n", - "|[2.4522765E-4, 5.1859417E-4, 1.7468519E-4, 2.262611E-4, 1.2124745E-4, 4.519367E-4, 3.961034E-5, 7.9740654E-5, 3.36356...|\n", - "|[8.990819E-5, 3.942771E-4, 1.682308E-4, 1.9284713E-4, 1.1525641E-4, 2.0554545E-4, 2.831427E-5, 1.4873769E-4, 3.316071...|\n", - "|[8.54337E-6, 3.4811173E-4, 5.7887002E-5, 4.8874565E-5, 5.402399E-5, 4.112481E-5, 1.4895271E-5, 6.21633E-4, 2.3461927E...|\n", - "|[2.1293292E-5, 2.3748633E-4, 9.990328E-5, 7.8908284E-5, 7.079569E-5, 1.2173247E-4, 7.88914E-6, 2.6103947E-4, 1.563249...|\n", - "|[8.190444E-5, 3.168154E-4, 1.590983E-4, 2.3166533E-4, 9.556332E-5, 1.9129038E-4, 1.7210512E-5, 9.8816345E-5, 1.854863...|\n", - "|[6.291183E-5, 2.6241466E-4, 1.054902E-4, 2.0039888E-4, 6.793013E-5, 2.0695888E-4, 1.1183583E-5, 8.1747305E-5, 1.71265...|\n", - "|[1.5287477E-4, 3.303019E-4, 7.293286E-5, 1.3093611E-4, 6.1460145E-5, 3.5471437E-4, 1.069333E-5, 6.7022884E-5, 2.26818...|\n", - "|[7.34195E-5, 3.2822473E-4, 1.6043369E-4, 1.9740193E-4, 1.036919E-4, 2.0666287E-4, 1.9626465E-5, 1.2458269E-4, 1.46395...|\n", - "|[5.8976515E-5, 3.6673344E-4, 9.322759E-5, 1.222245E-4, 7.065327E-5, 2.5420502E-4, 1.145865E-5, 2.2733102E-4, 2.255583...|\n", + "|[1.2838157E-4, 2.442499E-4, 6.756602E-5, 1.223822E-4, 5.718728E-5, 3.9370774E-4, 6.9826538E-6, 4.180329E-5, 1.21474E-...|\n", + "|[4.3975022E-5, 3.5182733E-4, 4.6756446E-5, 8.051952E-5, 3.157192E-5, 1.8915786E-4, 7.8848925E-6, 1.3820908E-4, 1.9617...|\n", + "|[1.0483801E-4, 2.2482511E-4, 2.9800098E-5, 6.471683E-5, 2.3306355E-5, 3.6853546E-4, 3.2802545E-6, 2.2436941E-5, 9.655...|\n", + "|[2.0184121E-5, 2.2646098E-4, 7.754879E-5, 6.9126E-5, 4.6796213E-5, 9.757494E-5, 5.5280707E-6, 2.3486002E-4, 1.3758638...|\n", + "|[1.1207414E-4, 2.3036542E-4, 5.2748997E-5, 1.0843094E-4, 3.9970357E-5, 3.692824E-4, 5.5317682E-6, 3.467135E-5, 1.1321...|\n", + "|[9.028466E-5, 2.0533502E-4, 4.5085282E-5, 7.65107E-5, 3.217092E-5, 3.3741904E-4, 3.8024857E-6, 4.1927728E-5, 9.920564...|\n", + "|[1.0625615E-4, 3.759827E-4, 7.6174496E-5, 1.2342798E-4, 4.7335903E-5, 3.3091815E-4, 1.0598523E-5, 9.161089E-5, 1.7926...|\n", + "|[2.2157477E-5, 2.726377E-4, 3.831429E-5, 6.2276886E-5, 1.8050652E-5, 1.7177712E-4, 6.0331595E-6, 1.06755506E-4, 1.790...|\n", + "|[1.0993216E-4, 2.8824335E-4, 4.2543048E-5, 1.06903855E-4, 3.039875E-5, 4.7743318E-4, 6.441006E-6, 3.6423717E-5, 1.361...|\n", + "|[9.6276366E-5, 2.047977E-4, 7.4698546E-5, 1.128771E-4, 4.6044628E-5, 2.8445767E-4, 5.6014956E-6, 5.475251E-5, 9.63856...|\n", + "|[7.3160336E-5, 3.2700456E-4, 1.3447899E-4, 1.7689951E-4, 8.4440886E-5, 2.2350134E-4, 1.3515168E-5, 1.1746432E-4, 1.81...|\n", + "|[8.632592E-5, 2.7143923E-4, 3.583003E-5, 7.763873E-5, 2.3417528E-5, 3.6477615E-4, 3.527159E-6, 3.646688E-5, 1.0721673...|\n", + "|[9.640316E-5, 2.7391897E-4, 5.7131063E-5, 1.09568326E-4, 3.8045353E-5, 3.472495E-4, 6.057242E-6, 4.3799748E-5, 1.1118...|\n", + "|[6.912533E-5, 2.5222785E-4, 5.0288483E-5, 1.1415517E-4, 2.9881658E-5, 2.7816373E-4, 4.972507E-6, 5.121496E-5, 1.15293...|\n", + "|[4.189945E-5, 2.4779947E-4, 1.2303083E-4, 1.4200866E-4, 7.2787174E-5, 1.600041E-4, 7.901948E-6, 1.3503798E-4, 1.46427...|\n", + "|[2.7033573E-5, 3.8410365E-4, 1.2880778E-4, 1.5630701E-4, 7.2431474E-5, 8.455686E-5, 1.2551222E-5, 1.9146077E-4, 2.293...|\n", + "|[2.9902518E-5, 3.521676E-4, 1.6034822E-4, 2.1348803E-4, 8.053424E-5, 1.00774814E-4, 1.3777179E-5, 1.5595586E-4, 1.615...|\n", + "|[3.2834323E-5, 2.8044736E-4, 1.8003663E-4, 2.017913E-4, 1.3718085E-4, 1.0062256E-4, 3.4619785E-5, 3.8973117E-4, 3.187...|\n", + "|[4.4552748E-5, 2.8623734E-4, 2.3419394E-4, 2.4108509E-4, 1.1926766E-4, 1.3529808E-4, 1.6018543E-5, 2.210266E-4, 1.558...|\n", + "|[1.2160183E-4, 2.8021698E-4, 6.289166E-5, 1.0147789E-4, 4.3161614E-5, 3.8964444E-4, 8.174407E-6, 6.2043844E-5, 1.5228...|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n", - "CPU times: user 18.5 ms, sys: 11.6 ms, total: 30.1 ms\n", - "Wall time: 6.85 s\n" + "CPU times: user 4.79 ms, sys: 1.93 ms, total: 6.72 ms\n", + "Wall time: 3.06 s\n" ] }, { @@ -972,7 +1029,7 @@ }, { "cell_type": "code", - "execution_count": 34, + "execution_count": 37, "id": "5f66d468-e0b1-4589-8606-b3848063a823", "metadata": {}, "outputs": [ @@ -990,31 +1047,31 @@ "+------------------------------------------------------------------------------------------------------------------------+\n", "| prediction|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", - "|[1.7042673E-4, 3.267522E-4, 7.581915E-5, 1.9806583E-4, 6.629557E-5, 4.2843417E-4, 1.31030665E-5, 5.2828378E-5, 2.3425...|\n", - "|[7.658893E-5, 5.898446E-4, 1.3134143E-4, 1.7956285E-4, 1.2959713E-4, 2.2623697E-4, 5.0602717E-5, 2.387755E-4, 3.85566...|\n", - "|[2.3643051E-5, 3.3990925E-4, 1.8499317E-4, 1.6320233E-4, 1.4257104E-4, 8.804509E-5, 2.6084534E-5, 2.3692146E-4, 1.479...|\n", - "|[7.265595E-5, 2.660495E-4, 1.2043192E-4, 2.0687138E-4, 8.6514396E-5, 2.2079627E-4, 1.3474356E-5, 8.564823E-5, 1.47982...|\n", - "|[5.583156E-5, 2.5246604E-4, 1.3152408E-4, 1.5752943E-4, 8.486259E-5, 2.1114835E-4, 9.902404E-6, 1.6352077E-4, 1.25058...|\n", - "|[1.1620977E-4, 2.9887763E-4, 9.283447E-5, 1.6428927E-4, 6.8689915E-5, 4.8077543E-4, 8.824636E-6, 5.926726E-5, 1.70530...|\n", - "|[1.1755548E-4, 2.1347901E-4, 5.9497E-5, 1.1462011E-4, 5.154583E-5, 4.378169E-4, 4.606332E-6, 3.3860095E-5, 8.858123E-...|\n", - "|[1.0416699E-4, 2.7650507E-4, 9.4154115E-5, 1.6408198E-4, 7.396577E-5, 4.2134034E-4, 8.686047E-6, 5.308059E-5, 1.27552...|\n", - "|[7.3212854E-5, 3.3290745E-4, 7.060745E-5, 1.7570716E-4, 7.5938726E-5, 2.7863326E-4, 1.5730891E-5, 7.8610516E-5, 1.641...|\n", - "|[5.3438263E-5, 3.4120653E-4, 4.2561773E-5, 6.7946996E-5, 4.2590156E-5, 3.6005717E-4, 4.079908E-6, 1.1698964E-4, 1.331...|\n", - "|[1.6286886E-5, 3.0098078E-4, 1.0513259E-4, 1.0939782E-4, 6.222565E-5, 8.1393446E-5, 8.843026E-6, 1.9695786E-4, 9.9110...|\n", - "|[2.4522765E-4, 5.1859417E-4, 1.7468519E-4, 2.262611E-4, 1.2124745E-4, 4.519367E-4, 3.961034E-5, 7.9740654E-5, 3.36356...|\n", - "|[8.990819E-5, 3.942771E-4, 1.682308E-4, 1.9284713E-4, 1.1525641E-4, 2.0554545E-4, 2.831427E-5, 1.4873769E-4, 3.316071...|\n", - "|[8.54337E-6, 3.4811173E-4, 5.7887002E-5, 4.8874565E-5, 5.402399E-5, 4.112481E-5, 1.4895271E-5, 6.21633E-4, 2.3461927E...|\n", - "|[2.1293292E-5, 2.3748633E-4, 9.990328E-5, 7.8908284E-5, 7.079569E-5, 1.2173247E-4, 7.88914E-6, 2.6103947E-4, 1.563249...|\n", - "|[8.190444E-5, 3.168154E-4, 1.590983E-4, 2.3166533E-4, 9.556332E-5, 1.9129038E-4, 1.7210512E-5, 9.8816345E-5, 1.854863...|\n", - "|[6.291183E-5, 2.6241466E-4, 1.054902E-4, 2.0039888E-4, 6.793013E-5, 2.0695888E-4, 1.1183583E-5, 8.1747305E-5, 1.71265...|\n", - "|[1.5287477E-4, 3.303019E-4, 7.293286E-5, 1.3093611E-4, 6.1460145E-5, 3.5471437E-4, 1.069333E-5, 6.7022884E-5, 2.26818...|\n", - "|[7.34195E-5, 3.2822473E-4, 1.6043369E-4, 1.9740193E-4, 1.036919E-4, 2.0666287E-4, 1.9626465E-5, 1.2458269E-4, 1.46395...|\n", - "|[5.8976515E-5, 3.6673344E-4, 9.322759E-5, 1.222245E-4, 7.065327E-5, 2.5420502E-4, 1.145865E-5, 2.2733102E-4, 2.255583...|\n", + "|[1.2838157E-4, 2.442499E-4, 6.756602E-5, 1.223822E-4, 5.718728E-5, 3.9370774E-4, 6.9826538E-6, 4.180329E-5, 1.21474E-...|\n", + "|[4.3975022E-5, 3.5182733E-4, 4.6756446E-5, 8.051952E-5, 3.157192E-5, 1.8915786E-4, 7.8848925E-6, 1.3820908E-4, 1.9617...|\n", + "|[1.0483801E-4, 2.2482511E-4, 2.9800098E-5, 6.471683E-5, 2.3306355E-5, 3.6853546E-4, 3.2802545E-6, 2.2436941E-5, 9.655...|\n", + "|[2.0184121E-5, 2.2646098E-4, 7.754879E-5, 6.9126E-5, 4.6796213E-5, 9.757494E-5, 5.5280707E-6, 2.3486002E-4, 1.3758638...|\n", + "|[1.1207414E-4, 2.3036542E-4, 5.2748997E-5, 1.0843094E-4, 3.9970357E-5, 3.692824E-4, 5.5317682E-6, 3.467135E-5, 1.1321...|\n", + "|[9.028466E-5, 2.0533502E-4, 4.5085282E-5, 7.65107E-5, 3.217092E-5, 3.3741904E-4, 3.8024857E-6, 4.1927728E-5, 9.920564...|\n", + "|[1.0625615E-4, 3.759827E-4, 7.6174496E-5, 1.2342798E-4, 4.7335903E-5, 3.3091815E-4, 1.0598523E-5, 9.161089E-5, 1.7926...|\n", + "|[2.2157477E-5, 2.726377E-4, 3.831429E-5, 6.2276886E-5, 1.8050652E-5, 1.7177712E-4, 6.0331595E-6, 1.06755506E-4, 1.790...|\n", + "|[1.0993216E-4, 2.8824335E-4, 4.2543048E-5, 1.06903855E-4, 3.039875E-5, 4.7743318E-4, 6.441006E-6, 3.6423717E-5, 1.361...|\n", + "|[9.6276366E-5, 2.047977E-4, 7.4698546E-5, 1.128771E-4, 4.6044628E-5, 2.8445767E-4, 5.6014956E-6, 5.475251E-5, 9.63856...|\n", + "|[7.3160336E-5, 3.2700456E-4, 1.3447899E-4, 1.7689951E-4, 8.4440886E-5, 2.2350134E-4, 1.3515168E-5, 1.1746432E-4, 1.81...|\n", + "|[8.632592E-5, 2.7143923E-4, 3.583003E-5, 7.763873E-5, 2.3417528E-5, 3.6477615E-4, 3.527159E-6, 3.646688E-5, 1.0721673...|\n", + "|[9.640316E-5, 2.7391897E-4, 5.7131063E-5, 1.09568326E-4, 3.8045353E-5, 3.472495E-4, 6.057242E-6, 4.3799748E-5, 1.1118...|\n", + "|[6.912533E-5, 2.5222785E-4, 5.0288483E-5, 1.1415517E-4, 2.9881658E-5, 2.7816373E-4, 4.972507E-6, 5.121496E-5, 1.15293...|\n", + "|[4.189945E-5, 2.4779947E-4, 1.2303083E-4, 1.4200866E-4, 7.2787174E-5, 1.600041E-4, 7.901948E-6, 1.3503798E-4, 1.46427...|\n", + "|[2.7033573E-5, 3.8410365E-4, 1.2880778E-4, 1.5630701E-4, 7.2431474E-5, 8.455686E-5, 1.2551222E-5, 1.9146077E-4, 2.293...|\n", + "|[2.9902518E-5, 3.521676E-4, 1.6034822E-4, 2.1348803E-4, 8.053424E-5, 1.00774814E-4, 1.3777179E-5, 1.5595586E-4, 1.615...|\n", + "|[3.2834323E-5, 2.8044736E-4, 1.8003663E-4, 2.017913E-4, 1.3718085E-4, 1.0062256E-4, 3.4619785E-5, 3.8973117E-4, 3.187...|\n", + "|[4.4552748E-5, 2.8623734E-4, 2.3419394E-4, 2.4108509E-4, 1.1926766E-4, 1.3529808E-4, 1.6018543E-5, 2.210266E-4, 1.558...|\n", + "|[1.2160183E-4, 2.8021698E-4, 6.289166E-5, 1.0147789E-4, 4.3161614E-5, 3.8964444E-4, 8.174407E-6, 6.2043844E-5, 1.5228...|\n", "+------------------------------------------------------------------------------------------------------------------------+\n", "only showing top 20 rows\n", "\n", - "CPU times: user 5.48 ms, sys: 4.51 ms, total: 9.99 ms\n", - "Wall time: 3.9 s\n" + "CPU times: user 3.16 ms, sys: 3.36 ms, total: 6.52 ms\n", + "Wall time: 2.24 s\n" ] }, { @@ -1033,7 +1090,7 @@ }, { "cell_type": "code", - "execution_count": 35, + "execution_count": 38, "id": "632c4c3a-fa52-4c3d-b71e-7526286e353a", "metadata": {}, "outputs": [ @@ -1048,8 +1105,8 @@ "name": "stdout", "output_type": "stream", "text": [ - "CPU times: user 18.8 ms, sys: 6.13 ms, total: 25 ms\n", - "Wall time: 22.2 s\n" + "CPU times: user 7.57 ms, sys: 5.2 ms, total: 12.8 ms\n", + "Wall time: 13.3 s\n" ] }, { @@ -1078,7 +1135,7 @@ }, { "cell_type": "code", - "execution_count": 36, + "execution_count": 39, "id": "bbfcaa51-3b9f-43ff-a4a8-4b46766115b8", "metadata": {}, "outputs": [ @@ -1095,7 +1152,7 @@ "[True]" ] }, - "execution_count": 36, + "execution_count": 39, "metadata": {}, "output_type": "execute_result" } @@ -1119,7 +1176,7 @@ }, { "cell_type": "code", - "execution_count": 37, + "execution_count": 40, "id": "0d88639b-d934-4eb4-ae2f-cc13b9b10456", "metadata": {}, "outputs": [], @@ -1138,7 +1195,7 @@ ], "metadata": { "kernelspec": { - "display_name": "Python 3 (ipykernel)", + "display_name": "spark-dl-tf", "language": "python", "name": "python3" }, @@ -1152,7 +1209,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.9.10" + "version": "3.11.9" } }, "nbformat": 4, diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/models_config/feature_columns/1/model.py b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/models_config/feature_columns/1/model.py index aa251e3d0..a2fa9635a 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/models_config/feature_columns/1/model.py +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/models_config/feature_columns/1/model.py @@ -69,7 +69,7 @@ def initialize(self, args): # Memory growth must be set before GPUs have been initialized print(e) - self.model = tf.keras.models.load_model("/my_pet_classifier") + self.model = tf.keras.models.load_model("/my_pet_classifier.keras") # You must parse model_config. JSON string is not parsed here self.model_config = model_config = json.loads(args['model_config']) @@ -134,7 +134,7 @@ def identity(input_tensor): # Get input numpy inputs = {name: transform(pb_utils.get_input_tensor_by_name(request, name)) for name, transform in input_transforms.items()} - pred = self.model.predict(inputs) + pred = self.model.predict(inputs, verbose=0) # Create output tensors. You need pb_utils.Tensor # objects to create pb_utils.InferenceResponse. diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/models_config/text_classification/1/model.py b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/models_config/text_classification/1/model.py index 50afea521..1bdef0b9c 100644 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/models_config/text_classification/1/model.py +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/models_config/text_classification/1/model.py @@ -26,6 +26,7 @@ import numpy as np import json +import tensorflow as tf # triton_python_backend_utils is available in every Triton Python model. You # need to use this module to create inference requests and responses. It also @@ -57,7 +58,6 @@ def initialize(self, args): """ import re import string - import tensorflow as tf from tensorflow.keras import layers print("tf: {}".format(tf.__version__)) @@ -83,7 +83,7 @@ def custom_standardization(input_data): "custom_standardization": custom_standardization} with tf.keras.utils.custom_object_scope(custom_objects): self.model = tf.keras.models.load_model( - "/text_model" + "/text_model_cleaned.keras", compile=False ) # You must parse model_config. JSON string is not parsed here @@ -126,11 +126,12 @@ def execute(self, requests): for request in requests: # Get input numpy sentence_input = pb_utils.get_input_tensor_by_name(request, "sentence") - sentences = list(sentence_input.as_numpy()) + sentences = sentence_input.as_numpy() sentences = np.squeeze(sentences).tolist() sentences = [s.decode('utf-8') for s in sentences] + sentences = tf.convert_to_tensor(sentences) - pred = self.model.predict(sentences) + pred = self.model.predict(sentences, verbose=0) # Create output tensors. You need pb_utils.Tensor # objects to create pb_utils.InferenceResponse. diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/text_classification.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/text_classification.ipynb deleted file mode 100644 index ed1289b11..000000000 --- a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/text_classification.ipynb +++ /dev/null @@ -1,1638 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "2cd2accf-5877-4136-a243-7a33a13ce2b4", - "metadata": {}, - "source": [ - "# Pyspark TensorFlow Inference\n", - "\n", - "## Text classification\n", - "Based on: https://www.tensorflow.org/tutorials/keras/text_classification" - ] - }, - { - "cell_type": "code", - "execution_count": 1, - "id": "76f0f5df-502f-444e-b2ee-1122e1dea870", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "import re\n", - "import shutil\n", - "import string\n", - "\n", - "import matplotlib.pyplot as plt\n", - "import tensorflow as tf\n", - "from tensorflow.keras import layers, losses" - ] - }, - { - "cell_type": "code", - "execution_count": 2, - "id": "a364ad5f-b269-45b5-ab8b-d8f34fb642b7", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "2.12.0\n" - ] - } - ], - "source": [ - "print(tf.__version__)" - ] - }, - { - "cell_type": "code", - "execution_count": 3, - "id": "d229c1b6-3967-46b5-9ea8-68f4b42dd211", - "metadata": {}, - "outputs": [], - "source": [ - "url = \"https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\"\n", - "\n", - "dataset = tf.keras.utils.get_file(\n", - " \"aclImdb_v1\", url, untar=True, cache_dir=\".\", cache_subdir=\"\"\n", - ")\n", - "\n", - "dataset_dir = os.path.join(os.path.dirname(dataset), \"aclImdb\")" - ] - }, - { - "cell_type": "code", - "execution_count": 4, - "id": "1f8038ae-8bc1-46bf-ae4c-6da08886c473", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['imdbEr.txt', 'imdb.vocab', 'train', 'test', 'README']" - ] - }, - "execution_count": 4, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "os.listdir(dataset_dir)" - ] - }, - { - "cell_type": "code", - "execution_count": 5, - "id": "12faaa3f-3441-4361-b9eb-4317e8c2c2f7", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['urls_unsup.txt',\n", - " 'neg',\n", - " 'unsupBow.feat',\n", - " 'unsup',\n", - " 'urls_pos.txt',\n", - " 'urls_neg.txt',\n", - " 'labeledBow.feat',\n", - " 'pos']" - ] - }, - "execution_count": 5, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "train_dir = os.path.join(dataset_dir, \"train\")\n", - "os.listdir(train_dir)" - ] - }, - { - "cell_type": "code", - "execution_count": 6, - "id": "152cc0cc-65d0-4e17-9ee8-222390df45b5", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Rachel Griffiths writes and directs this award winning short film. A heartwarming story about coping with grief and cherishing the memory of those we've loved and lost. Although, only 15 minutes long, Griffiths manages to capture so much emotion and truth onto film in the short space of time. Bud Tingwell gives a touching performance as Will, a widower struggling to cope with his wife's death. Will is confronted by the harsh reality of loneliness and helplessness as he proceeds to take care of Ruth's pet cow, Tulip. The film displays the grief and responsibility one feels for those they have loved and lost. Good cinematography, great direction, and superbly acted. It will bring tears to all those who have lost a loved one, and survived.\n" - ] - } - ], - "source": [ - "sample_file = os.path.join(train_dir, \"pos/1181_9.txt\")\n", - "with open(sample_file) as f:\n", - " print(f.read())" - ] - }, - { - "cell_type": "code", - "execution_count": 7, - "id": "b2277f58-78c8-4a12-bc98-5103e7c81a35", - "metadata": {}, - "outputs": [], - "source": [ - "remove_dir = os.path.join(train_dir, \"unsup\")\n", - "shutil.rmtree(remove_dir)" - ] - }, - { - "cell_type": "code", - "execution_count": 8, - "id": "ed83de92-ebb3-4170-b2bf-25265c6a6942", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Found 25000 files belonging to 2 classes.\n", - "Using 20000 files for training.\n" - ] - } - ], - "source": [ - "batch_size = 32\n", - "seed = 42\n", - "\n", - "raw_train_ds = tf.keras.utils.text_dataset_from_directory(\n", - " \"aclImdb/train\",\n", - " batch_size=batch_size,\n", - " validation_split=0.2,\n", - " subset=\"training\",\n", - " seed=seed,\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 9, - "id": "57c30568-daa8-4b2b-b30a-577c984a8af5", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Review b'\"Pandemonium\" is a horror movie spoof that comes off more stupid than funny. Believe me when I tell you, I love comedies. Especially comedy spoofs. \"Airplane\", \"The Naked Gun\" trilogy, \"Blazing Saddles\", \"High Anxiety\", and \"Spaceballs\" are some of my favorite comedies that spoof a particular genre. \"Pandemonium\" is not up there with those films. Most of the scenes in this movie had me sitting there in stunned silence because the movie wasn\\'t all that funny. There are a few laughs in the film, but when you watch a comedy, you expect to laugh a lot more than a few times and that\\'s all this film has going for it. Geez, \"Scream\" had more laughs than this film and that was more of a horror film. How bizarre is that?

*1/2 (out of four)'\n", - "Label 0\n", - "Review b\"David Mamet is a very interesting and a very un-equal director. His first movie 'House of Games' was the one I liked best, and it set a series of films with characters whose perspective of life changes as they get into complicated situations, and so does the perspective of the viewer.

So is 'Homicide' which from the title tries to set the mind of the viewer to the usual crime drama. The principal characters are two cops, one Jewish and one Irish who deal with a racially charged area. The murder of an old Jewish shop owner who proves to be an ancient veteran of the Israeli Independence war triggers the Jewish identity in the mind and heart of the Jewish detective.

This is were the flaws of the film are the more obvious. The process of awakening is theatrical and hard to believe, the group of Jewish militants is operatic, and the way the detective eventually walks to the final violent confrontation is pathetic. The end of the film itself is Mamet-like smart, but disappoints from a human emotional perspective.

Joe Mantegna and William Macy give strong performances, but the flaws of the story are too evident to be easily compensated.\"\n", - "Label 0\n", - "Review b'Great documentary about the lives of NY firefighters during the worst terrorist attack of all time.. That reason alone is why this should be a must see collectors item.. What shocked me was not only the attacks, but the\"High Fat Diet\" and physical appearance of some of these firefighters. I think a lot of Doctors would agree with me that,in the physical shape they were in, some of these firefighters would NOT of made it to the 79th floor carrying over 60 lbs of gear. Having said that i now have a greater respect for firefighters and i realize becoming a firefighter is a life altering job. The French have a history of making great documentary\\'s and that is what this is, a Great Documentary.....'\n", - "Label 1\n" - ] - } - ], - "source": [ - "for text_batch, label_batch in raw_train_ds.take(1):\n", - " for i in range(3):\n", - " print(\"Review\", text_batch.numpy()[i])\n", - " print(\"Label\", label_batch.numpy()[i])" - ] - }, - { - "cell_type": "code", - "execution_count": 10, - "id": "1e863eb6-4bd7-4da0-b10d-d951b5ee52bd", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Label 0 corresponds to neg\n", - "Label 1 corresponds to pos\n" - ] - } - ], - "source": [ - "print(\"Label 0 corresponds to\", raw_train_ds.class_names[0])\n", - "print(\"Label 1 corresponds to\", raw_train_ds.class_names[1])" - ] - }, - { - "cell_type": "code", - "execution_count": 11, - "id": "1593e2e5-df51-4fbf-b4be-c786e740ddab", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Found 25000 files belonging to 2 classes.\n", - "Using 5000 files for validation.\n" - ] - } - ], - "source": [ - "raw_val_ds = tf.keras.utils.text_dataset_from_directory(\n", - " \"aclImdb/train\",\n", - " batch_size=batch_size,\n", - " validation_split=0.2,\n", - " subset=\"validation\",\n", - " seed=seed,\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 12, - "id": "944fd61d-3926-4296-889a-b2a375a1b039", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Found 25000 files belonging to 2 classes.\n" - ] - } - ], - "source": [ - "raw_test_ds = tf.keras.utils.text_dataset_from_directory(\n", - " \"aclImdb/test\", batch_size=batch_size\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 13, - "id": "cb141709-fcc1-4cee-bc98-9c89aaba8648", - "metadata": {}, - "outputs": [], - "source": [ - "def custom_standardization(input_data):\n", - " lowercase = tf.strings.lower(input_data)\n", - " stripped_html = tf.strings.regex_replace(lowercase, \"
\", \" \")\n", - " return tf.strings.regex_replace(\n", - " stripped_html, \"[%s]\" % re.escape(string.punctuation), \"\"\n", - " )" - ] - }, - { - "cell_type": "code", - "execution_count": 14, - "id": "d4e80ea9-536a-4ebc-8b35-1eca73dbba7d", - "metadata": {}, - "outputs": [], - "source": [ - "max_features = 10000\n", - "sequence_length = 250\n", - "\n", - "vectorize_layer = layers.TextVectorization(\n", - " standardize=custom_standardization,\n", - " max_tokens=max_features,\n", - " output_mode=\"int\",\n", - " output_sequence_length=sequence_length,\n", - ")" - ] - }, - { - "cell_type": "code", - "execution_count": 15, - "id": "ad1e5d81-7dae-4b08-b520-ca45501b9510", - "metadata": {}, - "outputs": [], - "source": [ - "# Make a text-only dataset (without labels), then call adapt\n", - "train_text = raw_train_ds.map(lambda x, y: x)\n", - "vectorize_layer.adapt(train_text)" - ] - }, - { - "cell_type": "code", - "execution_count": 16, - "id": "80f243f5-edd3-4e1c-bddc-abc1cc6673ef", - "metadata": {}, - "outputs": [], - "source": [ - "def vectorize_text(text, label):\n", - " text = tf.expand_dims(text, -1)\n", - " return vectorize_layer(text), label" - ] - }, - { - "cell_type": "code", - "execution_count": 17, - "id": "8f37e95c-515c-4edb-a1ee-fc47be5df4b9", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Review tf.Tensor(b'Great movie - especially the music - Etta James - \"At Last\". This speaks volumes when you have finally found that special someone.', shape=(), dtype=string)\n", - "Label neg\n", - "Vectorized review (, )\n" - ] - } - ], - "source": [ - "# retrieve a batch (of 32 reviews and labels) from the dataset\n", - "text_batch, label_batch = next(iter(raw_train_ds))\n", - "first_review, first_label = text_batch[0], label_batch[0]\n", - "print(\"Review\", first_review)\n", - "print(\"Label\", raw_train_ds.class_names[first_label])\n", - "print(\"Vectorized review\", vectorize_text(first_review, first_label))" - ] - }, - { - "cell_type": "code", - "execution_count": 18, - "id": "60c9208a-39ac-4e6c-a603-61038cdf3d10", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "1287 ---> silent\n", - " 313 ---> night\n", - "Vocabulary size: 10000\n" - ] - } - ], - "source": [ - "print(\"1287 ---> \",vectorize_layer.get_vocabulary()[1287])\n", - "print(\" 313 ---> \",vectorize_layer.get_vocabulary()[313])\n", - "print('Vocabulary size: {}'.format(len(vectorize_layer.get_vocabulary())))" - ] - }, - { - "cell_type": "code", - "execution_count": 19, - "id": "3cf90d4b-8dae-44b2-b32b-80cb0092c430", - "metadata": {}, - "outputs": [], - "source": [ - "train_ds = raw_train_ds.map(vectorize_text)\n", - "val_ds = raw_val_ds.map(vectorize_text)\n", - "test_ds = raw_test_ds.map(vectorize_text)" - ] - }, - { - "cell_type": "code", - "execution_count": 20, - "id": "115a5aba-8a00-458f-be25-0aae9f55de22", - "metadata": {}, - "outputs": [], - "source": [ - "AUTOTUNE = tf.data.AUTOTUNE\n", - "\n", - "train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)\n", - "val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)\n", - "test_ds = test_ds.cache().prefetch(buffer_size=AUTOTUNE)" - ] - }, - { - "cell_type": "code", - "execution_count": 21, - "id": "d64f4495-102d-4244-9b42-1ba9976a366e", - "metadata": {}, - "outputs": [], - "source": [ - "embedding_dim = 16" - ] - }, - { - "cell_type": "code", - "execution_count": 22, - "id": "3dc95d22-935f-4091-b0ee-da95174eb9a0", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Model: \"sequential\"\n", - "_________________________________________________________________\n", - " Layer (type) Output Shape Param # \n", - "=================================================================\n", - " embedding (Embedding) (None, None, 16) 160016 \n", - " \n", - " dropout (Dropout) (None, None, 16) 0 \n", - " \n", - " global_average_pooling1d (G (None, 16) 0 \n", - " lobalAveragePooling1D) \n", - " \n", - " dropout_1 (Dropout) (None, 16) 0 \n", - " \n", - " dense (Dense) (None, 1) 17 \n", - " \n", - "=================================================================\n", - "Total params: 160,033\n", - "Trainable params: 160,033\n", - "Non-trainable params: 0\n", - "_________________________________________________________________\n" - ] - } - ], - "source": [ - "model = tf.keras.Sequential([\n", - " layers.Embedding(max_features + 1, embedding_dim),\n", - " layers.Dropout(0.2),\n", - " layers.GlobalAveragePooling1D(),\n", - " layers.Dropout(0.2),\n", - " layers.Dense(1)])\n", - "\n", - "model.summary()" - ] - }, - { - "cell_type": "code", - "execution_count": 23, - "id": "d9059b93-7666-46db-bf15-517c4c205df9", - "metadata": {}, - "outputs": [], - "source": [ - "model.compile(loss=losses.BinaryCrossentropy(from_logits=True),\n", - " optimizer='adam',\n", - " metrics=tf.metrics.BinaryAccuracy(threshold=0.0))" - ] - }, - { - "cell_type": "code", - "execution_count": 24, - "id": "b1d5959f-1bd8-48da-9815-8239599519b2", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Epoch 1/10\n", - "625/625 [==============================] - 5s 7ms/step - loss: 0.6647 - binary_accuracy: 0.6946 - val_loss: 0.6154 - val_binary_accuracy: 0.7736\n", - "Epoch 2/10\n", - "625/625 [==============================] - 4s 7ms/step - loss: 0.5483 - binary_accuracy: 0.8021 - val_loss: 0.4979 - val_binary_accuracy: 0.8232\n", - "Epoch 3/10\n", - "625/625 [==============================] - 4s 7ms/step - loss: 0.4444 - binary_accuracy: 0.8442 - val_loss: 0.4200 - val_binary_accuracy: 0.8468\n", - "Epoch 4/10\n", - "625/625 [==============================] - 4s 7ms/step - loss: 0.3781 - binary_accuracy: 0.8654 - val_loss: 0.3738 - val_binary_accuracy: 0.8598\n", - "Epoch 5/10\n", - "625/625 [==============================] - 4s 7ms/step - loss: 0.3351 - binary_accuracy: 0.8786 - val_loss: 0.3447 - val_binary_accuracy: 0.8674\n", - "Epoch 6/10\n", - "625/625 [==============================] - 4s 7ms/step - loss: 0.3054 - binary_accuracy: 0.8877 - val_loss: 0.3261 - val_binary_accuracy: 0.8704\n", - "Epoch 7/10\n", - "625/625 [==============================] - 4s 7ms/step - loss: 0.2802 - binary_accuracy: 0.8977 - val_loss: 0.3128 - val_binary_accuracy: 0.8734\n", - "Epoch 8/10\n", - "625/625 [==============================] - 4s 7ms/step - loss: 0.2616 - binary_accuracy: 0.9056 - val_loss: 0.3031 - val_binary_accuracy: 0.8756\n", - "Epoch 9/10\n", - "625/625 [==============================] - 4s 7ms/step - loss: 0.2461 - binary_accuracy: 0.9111 - val_loss: 0.2965 - val_binary_accuracy: 0.8778\n", - "Epoch 10/10\n", - "625/625 [==============================] - 4s 7ms/step - loss: 0.2309 - binary_accuracy: 0.9170 - val_loss: 0.2918 - val_binary_accuracy: 0.8794\n" - ] - } - ], - "source": [ - "epochs = 10\n", - "history = model.fit(\n", - " train_ds,\n", - " validation_data=val_ds,\n", - " epochs=epochs)" - ] - }, - { - "cell_type": "code", - "execution_count": 25, - "id": "656afe07-354f-4ff2-8e3e-d02bad6c5958", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "782/782 [==============================] - 3s 3ms/step - loss: 0.3103 - binary_accuracy: 0.8737\n", - "Loss: 0.3103351294994354\n", - "Accuracy: 0.8736799955368042\n" - ] - } - ], - "source": [ - "loss, accuracy = model.evaluate(test_ds)\n", - "\n", - "print(\"Loss: \", loss)\n", - "print(\"Accuracy: \", accuracy)" - ] - }, - { - "cell_type": "code", - "execution_count": 26, - "id": "a01d0f13-d0b8-4d78-9ddc-ede5ed402446", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "dict_keys(['loss', 'binary_accuracy', 'val_loss', 'val_binary_accuracy'])" - ] - }, - "execution_count": 26, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "history_dict = history.history\n", - "history_dict.keys()" - ] - }, - { - "cell_type": "code", - "execution_count": 27, - "id": "1f7484c3-3cdf-46d5-b95d-80316f0e6240", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABVeklEQVR4nO3deZyNdf/H8deZGbMZM/ZZGGNfs2W7cVsqhcqajCJLpZI16cZtpyhSthB3KJVEQyq7KKGbQkkSd7YwljBj7M5cvz+u3xxzmsXs18w57+fjcR6uc53rXNfnmKnz9v1+r+/XZhiGgYiIiIiL8LC6ABEREZGspHAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjYoGePXtSunTpDL137Nix2Gy2rC0olzl69Cg2m41Fixbl6HW3bNmCzWZjy5Ytjn1p/VllV82lS5emZ8+eWXrOtFi0aBE2m42jR4/m+LVFMkvhRiQRm82WpkfiLz+RzNq+fTtjx47l0qVLVpci4hK8rC5AJDdZvHix0/MPPviADRs2JNlfpUqVTF1n/vz5xMfHZ+i9I0eOZNiwYZm6vqRdZn5WabV9+3bGjRtHz549KViwoNNrBw8exMND/w4VSQ+FG5FEunXr5vT8+++/Z8OGDUn2/93Vq1fx9/dP83Xy5cuXofoAvLy88PLSf7o5JTM/q6zg4+Nj6fVF8iL9c0AknZo3b84999zDjz/+SNOmTfH39+ff//43AJ9//jmPPPIIYWFh+Pj4UK5cOSZMmIDdbnc6x9/HcSSM13jzzTeZN28e5cqVw8fHh3r16rFr1y6n9yY35sZms9GvXz9WrlzJPffcg4+PD9WqVWPt2rVJ6t+yZQt169bF19eXcuXK8e6776Z5HM/WrVt5/PHHKVWqFD4+PoSHh/PSSy9x7dq1JJ8vICCAkydP0r59ewICAihWrBhDhgxJ8ndx6dIlevbsSVBQEAULFqRHjx5p6p754YcfsNlsvP/++0leW7duHTabjS+//BKAY8eO8eKLL1KpUiX8/PwoUqQIjz/+eJrGkyQ35iatNf/888/07NmTsmXL4uvrS0hICE8//TR//fWX45ixY8fyyiuvAFCmTBlH12dCbcmNufnjjz94/PHHKVy4MP7+/vzjH//gq6++cjomYfzQp59+ymuvvUbJkiXx9fXlgQce4PDhw3f93CmZPXs21apVw8fHh7CwMPr27Zvksx86dIjHHnuMkJAQfH19KVmyJF26dCEmJsZxzIYNG/jnP/9JwYIFCQgIoFKlSo7/jkQyS//8E8mAv/76i9atW9OlSxe6detGcHAwYA7CDAgIYPDgwQQEBPD1118zevRoYmNjmTJlyl3P+/HHH3P58mWef/55bDYbkydPpmPHjvzxxx93bUH47rvviIqK4sUXX6RAgQLMmDGDxx57jOPHj1OkSBEA9uzZQ6tWrQgNDWXcuHHY7XbGjx9PsWLF0vS5ly1bxtWrV+nTpw9FihRh586dzJw5kz///JNly5Y5HWu322nZsiUNGjTgzTffZOPGjUydOpVy5crRp08fAAzDoF27dnz33Xe88MILVKlShRUrVtCjR4+71lK3bl3Kli3Lp59+muT4pUuXUqhQIVq2bAnArl272L59O126dKFkyZIcPXqUOXPm0Lx5c3799dd0tbqlp+YNGzbwxx9/0KtXL0JCQti/fz/z5s1j//79fP/999hsNjp27Mjvv//OkiVLePvttylatChAij+TM2fO0KhRI65evcqAAQMoUqQI77//Pm3btmX58uV06NDB6fjXX38dDw8PhgwZQkxMDJMnT6Zr167897//TfNnTjB27FjGjRtHixYt6NOnDwcPHmTOnDns2rWLbdu2kS9fPm7evEnLli25ceMG/fv3JyQkhJMnT/Lll19y6dIlgoKC2L9/P48++ig1atRg/Pjx+Pj4cPjwYbZt25bumkSSZYhIivr27Wv8/T+TZs2aGYAxd+7cJMdfvXo1yb7nn3/e8Pf3N65fv+7Y16NHDyMiIsLx/MiRIwZgFClSxLhw4YJj/+eff24AxhdffOHYN2bMmCQ1AYa3t7dx+PBhx76ffvrJAIyZM2c69rVp08bw9/c3Tp486dh36NAhw8vLK8k5k5Pc55s0aZJhs9mMY8eOOX0+wBg/frzTsbVr1zbq1KnjeL5y5UoDMCZPnuzYd/v2baNJkyYGYCxcuDDVeoYPH27ky5fP6e/sxo0bRsGCBY2nn3461bp37NhhAMYHH3zg2Ld582YDMDZv3uz0WRL/rNJTc3LXXbJkiQEY3377rWPflClTDMA4cuRIkuMjIiKMHj16OJ4PGjTIAIytW7c69l2+fNkoU6aMUbp0acNutzt9lipVqhg3btxwHDt9+nQDMPbt25fkWoktXLjQqaazZ88a3t7exkMPPeS4hmEYxqxZswzAWLBggWEYhrFnzx4DMJYtW5biud9++20DMM6dO5dqDSIZpW4pkQzw8fGhV69eSfb7+fk5ti9fvsz58+dp0qQJV69e5bfffrvreSMjIylUqJDjeZMmTQCzG+JuWrRoQbly5RzPa9SoQWBgoOO9drudjRs30r59e8LCwhzHlS9fntatW9/1/OD8+a5cucL58+dp1KgRhmGwZ8+eJMe/8MILTs+bNGni9FlWr16Nl5eXoyUHwNPTk/79+6epnsjISG7dukVUVJRj3/r167l06RKRkZHJ1n3r1i3++usvypcvT8GCBdm9e3earpWRmhNf9/r165w/f55//OMfAOm+buLr169fn3/+85+OfQEBATz33HMcPXqUX3/91en4Xr164e3t7Xient+pxDZu3MjNmzcZNGiQ0wDn3r17ExgY6OgWCwoKAsyuwatXryZ7roRB059//nm2D9YW96RwI5IBJUqUcPrCSLB//346dOhAUFAQgYGBFCtWzDEYOfF4g5SUKlXK6XlC0Ll48WK635vw/oT3nj17lmvXrlG+fPkkxyW3LznHjx+nZ8+eFC5c2DGOplmzZkDSz+fr65ukayVxPWCOhQkNDSUgIMDpuEqVKqWpnpo1a1K5cmWWLl3q2Ld06VKKFi3K/fff79h37do1Ro8eTXh4OD4+PhQtWpRixYpx6dKlNP1cEktPzRcuXGDgwIEEBwfj5+dHsWLFKFOmDJC234eUrp/ctRLu4Dt27JjT/sz8Tv39upD0c3p7e1O2bFnH62XKlGHw4MH85z//oWjRorRs2ZJ33nnH6fNGRkbSuHFjnn32WYKDg+nSpQuffvqpgo5kGY25EcmAxP8iT3Dp0iWaNWtGYGAg48ePp1y5cvj6+rJ7926GDh2apv9xe3p6JrvfMIxsfW9a2O12HnzwQS5cuMDQoUOpXLky+fPn5+TJk/Ts2TPJ50upnqwWGRnJa6+9xvnz5ylQoACrVq3iiSeecLqjrH///ixcuJBBgwbRsGFDgoKCsNlsdOnSJVu/UDt37sz27dt55ZVXqFWrFgEBAcTHx9OqVasc+yLP7t+L5EydOpWePXvy+eefs379egYMGMCkSZP4/vvvKVmyJH5+fnz77bds3ryZr776irVr17J06VLuv/9+1q9fn2O/O+K6FG5EssiWLVv466+/iIqKomnTpo79R44csbCqO4oXL46vr2+yd8qk5e6Zffv28fvvv/P+++/TvXt3x/4NGzZkuKaIiAg2bdpEXFycU0vIwYMH03yOyMhIxo0bx2effUZwcDCxsbF06dLF6Zjly5fTo0cPpk6d6th3/fr1DE2al9aaL168yKZNmxg3bhyjR4927D906FCSc6ZnxumIiIhk/34Suj0jIiLSfK70SDjvwYMHKVu2rGP/zZs3OXLkCC1atHA6vnr16lSvXp2RI0eyfft2GjduzNy5c3n11VcB8PDw4IEHHuCBBx7grbfeYuLEiYwYMYLNmzcnOZdIeqlbSiSLJPxrM/G/iG/evMns2bOtKsmJp6cnLVq0YOXKlZw6dcqx//Dhw6xZsyZN7wfnz2cYBtOnT89wTQ8//DC3b99mzpw5jn12u52ZM2em+RxVqlShevXqLF26lKVLlxIaGuoULhNq/3tLxcyZM5Pclp6VNSf39wUwbdq0JOfMnz8/QJrC1sMPP8zOnTvZsWOHY9+VK1eYN28epUuXpmrVqmn9KOnSokULvL29mTFjhtNneu+994iJieGRRx4BIDY2ltu3bzu9t3r16nh4eHDjxg3A7K77u1q1agE4jhHJDLXciGSRRo0aUahQIXr06MGAAQOw2WwsXrw4W5v/02vs2LGsX7+exo0b06dPH+x2O7NmzeKee+5h7969qb63cuXKlCtXjiFDhnDy5EkCAwP57LPP0j12I7E2bdrQuHFjhg0bxtGjR6latSpRUVHpHo8SGRnJ6NGj8fX15Zlnnkkyo++jjz7K4sWLCQoKomrVquzYsYONGzc6bpHPjpoDAwNp2rQpkydP5tatW5QoUYL169cn25JXp04dAEaMGEGXLl3Ily8fbdq0cYSexIYNG8aSJUto3bo1AwYMoHDhwrz//vscOXKEzz77LNtmMy5WrBjDhw9n3LhxtGrVirZt23Lw4EFmz55NvXr1HGPLvv76a/r168fjjz9OxYoVuX37NosXL8bT05PHHnsMgPHjx/Ptt9/yyCOPEBERwdmzZ5k9ezYlS5Z0GigtklEKNyJZpEiRInz55Ze8/PLLjBw5kkKFCtGtWzceeOABx3wrVqtTpw5r1qxhyJAhjBo1ivDwcMaPH8+BAwfuejdXvnz5+OKLLxzjJ3x9fenQoQP9+vWjZs2aGarHw8ODVatWMWjQID788ENsNhtt27Zl6tSp1K5dO83niYyMZOTIkVy9etXpLqkE06dPx9PTk48++ojr16/TuHFjNm7cmKGfS3pq/vjjj+nfvz/vvPMOhmHw0EMPsWbNGqe71QDq1avHhAkTmDt3LmvXriU+Pp4jR44kG26Cg4PZvn07Q4cOZebMmVy/fp0aNWrwxRdfOFpPssvYsWMpVqwYs2bN4qWXXqJw4cI899xzTJw40TEPU82aNWnZsiVffPEFJ0+exN/fn5o1a7JmzRrHnWJt27bl6NGjLFiwgPPnz1O0aFGaNWvGuHHjHHdbiWSGzchN/6wUEUu0b9+e/fv3JzseREQkr9GYGxE38/elEg4dOsTq1atp3ry5NQWJiGQxtdyIuJnQ0FDHekfHjh1jzpw53Lhxgz179lChQgWryxMRyTSNuRFxM61atWLJkiVER0fj4+NDw4YNmThxooKNiLgMtdyIiIiIS9GYGxEREXEpCjciIiLiUtxuzE18fDynTp2iQIEC6ZryXERERKxjGAaXL18mLCzsrpNVul24OXXqFOHh4VaXISIiIhlw4sQJSpYsmeoxbhduChQoAJh/OYGBgRZXIyIiImkRGxtLeHi443s8NW4XbhK6ogIDAxVuRERE8pi0DCnRgGIRERFxKQo3IiIi4lIUbkRERMSluN2YGxERyVp2u51bt25ZXYa4AG9v77ve5p0WCjciIpIhhmEQHR3NpUuXrC5FXISHhwdlypTB29s7U+dRuBERkQxJCDbFixfH399fE6NKpiRMsnv69GlKlSqVqd8nhRsREUk3u93uCDZFihSxuhxxEcWKFePUqVPcvn2bfPnyZfg8GlAsIiLpljDGxt/f3+JKxJUkdEfZ7fZMnUfhRkREMkxdUZKVsur3Sd1SWcRuh61b4fRpCA2FJk3A09PqqkRERNyPWm6yQFQUlC4N990HTz5p/lm6tLlfRERcX+nSpZk2bVqaj9+yZQs2my3b7zRbtGgRBQsWzNZr5EYKN5kUFQWdOsGffzrvP3nS3K+AIyKSOrsdtmyBJUvMPzM53CJVNpst1cfYsWMzdN5du3bx3HPPpfn4Ro0acfr0aYKCgjJ0PUmduqUywW6HgQPBMJK+Zhhgs8GgQdCunbqoRESSExVl/n808T8QS5aE6dOhY8esv97p06cd20uXLmX06NEcPHjQsS8gIMCxbRgGdrsdL6+7f1UWK1YsXXV4e3sTEhKSrvdI2qnlJhO2bk3aYpOYYcCJE+ZxIiLizIqW75CQEMcjKCgIm83meP7bb79RoEAB1qxZQ506dfDx8eG7777jf//7H+3atSM4OJiAgADq1avHxo0bnc77924pm83Gf/7zHzp06IC/vz8VKlRg1apVjtf/3i2V0H20bt06qlSpQkBAAK1atXIKY7dv32bAgAEULFiQIkWKMHToUHr06EH79u3T9XcwZ84cypUrh7e3N5UqVWLx4sWO1wzDYOzYsZQqVQofHx/CwsIYMGCA4/XZs2dToUIFfH19CQ4OplOnTum6dk5RuMmERL9zWXKciIi7uFvLN5gt39nZRZWSYcOG8frrr3PgwAFq1KhBXFwcDz/8MJs2bWLPnj20atWKNm3acPz48VTPM27cODp37szPP//Mww8/TNeuXblw4UKKx1+9epU333yTxYsX8+2333L8+HGGDBnieP2NN97go48+YuHChWzbto3Y2FhWrlyZrs+2YsUKBg4cyMsvv8wvv/zC888/T69evdi8eTMAn332GW+//Tbvvvsuhw4dYuXKlVSvXh2AH374gQEDBjB+/HgOHjzI2rVradq0abqun2MMNxMTE2MARkxMTKbPtXmzYZj/Gab+2Lw505cSEclVrl27Zvz666/GtWvXMvT+3PD/z4ULFxpBQUGJatpsAMbKlSvv+t5q1aoZM2fOdDyPiIgw3n77bcdzwBg5cqTjeVxcnAEYa9ascbrWxYsXHbUAxuHDhx3veeedd4zg4GDH8+DgYGPKlCmO57dv3zZKlSpltGvXLs2fsVGjRkbv3r2djnn88ceNhx9+2DAMw5g6dapRsWJF4+bNm0nO9dlnnxmBgYFGbGxsitfLrNR+r9Lz/a2Wm0xo0sTsG07ptnybDcLDzeNEROSO3NzyXbduXafncXFxDBkyhCpVqlCwYEECAgI4cODAXVtuatSo4djOnz8/gYGBnD17NsXj/f39KVeunON5aGio4/iYmBjOnDlD/fr1Ha97enpSp06ddH22AwcO0LhxY6d9jRs35sCBAwA8/vjjXLt2jbJly9K7d29WrFjB7du3AXjwwQeJiIigbNmyPPXUU3z00UdcvXo1XdfPKQo3meDpaQ56g6QBJ+H5tGkaTCwi8nehoVl7XFbKnz+/0/MhQ4awYsUKJk6cyNatW9m7dy/Vq1fn5s2bqZ7n78sH2Gw24uPj03W8kVy/XTYKDw/n4MGDzJ49Gz8/P1588UWaNm3KrVu3KFCgALt372bJkiWEhoYyevRoatasmSsXTlW4yaSOHWH5cihRwnl/yZLm/uwY7S8iktflpZbvbdu20bNnTzp06ED16tUJCQnh6NGjOVpDUFAQwcHB7Nq1y7HPbreze/fudJ2nSpUqbNu2zWnftm3bqFq1quO5n58fbdq0YcaMGWzZsoUdO3awb98+ALy8vGjRogWTJ0/m559/5ujRo3z99deZ+GTZQ7eCZ4GOHc3bvTVDsYhI2iS0fHfqZAaZxA0Uua3lu0KFCkRFRdGmTRtsNhujRo1KtQUmu/Tv359JkyZRvnx5KleuzMyZM7l48WK6lix45ZVX6Ny5M7Vr16ZFixZ88cUXREVFOe7+WrRoEXa7nQYNGuDv78+HH36In58fERERfPnll/zxxx80bdqUQoUKsXr1auLj46lUqVJ2feQMU7jJIp6e0Ly51VWIiOQdCS3fyc1zM21a7mn5fuutt3j66adp1KgRRYsWZejQocTGxuZ4HUOHDiU6Opru3bvj6enJc889R8uWLfFMRwJs374906dP580332TgwIGUKVOGhQsX0vz/v8AKFizI66+/zuDBg7Hb7VSvXp0vvviCIkWKULBgQaKiohg7dizXr1+nQoUKLFmyhGrVqmXTJ844m5HTHXoWi42NJSgoiJiYGAIDA60uR0QkT7p+/TpHjhyhTJky+Pr6ZupcWpsvY+Lj46lSpQqdO3dmwoQJVpeTJVL7vUrP97dabkRExFJq+U6bY8eOsX79epo1a8aNGzeYNWsWR44c4cknn7S6tFxHA4pFRETyAA8PDxYtWkS9evVo3Lgx+/btY+PGjVSpUsXq0nIdtdyIiIjkAeHh4UnudJLkqeVGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lIUbkRERMSlKNyIiIiIS1G4ERERSafmzZszaNAgx/PSpUszbdq0VN9js9lYuXJlpq+dVedJzdixY6lVq1a2XiM7KdyIiIjbaNOmDa1atUr2ta1bt2Kz2fj555/Tfd5du3bx3HPPZbY8JykFjNOnT9O6dessvZarUbgRERG38cwzz7Bhwwb+TLxS5/9buHAhdevWpUaNGuk+b7FixfD398+KEu8qJCQEHx+fHLlWXqVwIyIibuPRRx+lWLFiLFq0yGl/XFwcy5Yt45lnnuGvv/7iiSeeoESJEvj7+1O9enWWLFmS6nn/3i116NAhmjZtiq+vL1WrVmXDhg1J3jN06FAqVqyIv78/ZcuWZdSoUdy6dQuARYsWMW7cOH766SdsNhs2m81R89+7pfbt28f999+Pn58fRYoU4bnnniMuLs7xes+ePWnfvj1vvvkmoaGhFClShL59+zqulRbx8fGMHz+ekiVL4uPjQ61atVi7dq3j9Zs3b9KvXz9CQ0Px9fUlIiKCSZMmAWAYBmPHjqVUqVL4+PgQFhbGgAED0nztjNDyCyIikiUMA65eteba/v5gs939OC8vL7p3786iRYsYMWIEtv9/07Jly7Db7TzxxBPExcVRp04dhg4dSmBgIF999RVPPfUU5cqVo379+ne9Rnx8PB07diQ4OJj//ve/xMTEOI3PSVCgQAEWLVpEWFgY+/bto3fv3hQoUIB//etfREZG8ssvv7B27Vo2btwIQFBQUJJzXLlyhZYtW9KwYUN27drF2bNnefbZZ+nXr59TgNu8eTOhoaFs3ryZw4cPExkZSa1atejdu/fd/9KA6dOnM3XqVN59911q167NggULaNu2Lfv376dChQrMmDGDVatW8emnn1KqVClOnDjBiRMnAPjss894++23+eSTT6hWrRrR0dH89NNPabpuhhluJiYmxgCMmJgYq0sREcmzrl27Zvz666/GtWvXHPvi4gzDjDg5/4iLS3vtBw4cMABj8+bNjn1NmjQxunXrluJ7HnnkEePll192PG/WrJkxcOBAx/OIiAjj7bffNgzDMNatW2d4eXkZJ0+edLy+Zs0aAzBWrFiR4jWmTJli1KlTx/F8zJgxRs2aNZMcl/g88+bNMwoVKmTEJfoL+OqrrwwPDw8jOjraMAzD6NGjhxEREWHcvn3bcczjjz9uREZGpljL368dFhZmvPbaa07H1KtXz3jxxRcNwzCM/v37G/fff78RHx+f5FxTp041KlasaNy8eTPF6yVI7vcqQXq+v9UtJSIibqVy5co0atSIBQsWAHD48GG2bt3KM888A4DdbmfChAlUr16dwoULExAQwLp16zh+/Hiazn/gwAHCw8MJCwtz7GvYsGGS45YuXUrjxo0JCQkhICCAkSNHpvkaia9Vs2ZN8ufP79jXuHFj4uPjOXjwoGNftWrV8PT0dDwPDQ3l7NmzabpGbGwsp06donHjxk77GzduzIEDBwCz62vv3r1UqlSJAQMGsH79esdxjz/+ONeuXaNs2bL07t2bFStWcPv27XR9zvRSuBERkSzh7w9xcdY80juW95lnnuGzzz7j8uXLLFy4kHLlytGsWTMApkyZwvTp0xk6dCibN29m7969tGzZkps3b2bZ39WOHTvo2rUrDz/8MF9++SV79uxhxIgRWXqNxPLly+f03GazER8fn2Xnv/feezly5AgTJkzg2rVrdO7cmU6dOgHmauYHDx5k9uzZ+Pn58eKLL9K0adN0jflJL425ERGRLGGzQaIGhFytc+fODBw4kI8//pgPPviAPn36OMbfbNu2jXbt2tGtWzfAHEPz+++/U7Vq1TSdu0qVKpw4cYLTp08TGhoKwPfff+90zPbt24mIiGDEiBGOfceOHXM6xtvbG7vdftdrLVq0iCtXrjhab7Zt24aHhweVKlVKU713ExgYSFhYGNu2bXMEwITrJB6DFBgYSGRkJJGRkXTq1IlWrVpx4cIFChcujJ+fH23atKFNmzb07duXypUrs2/fPu69994sqfHvFG5ERMTtBAQEEBkZyfDhw4mNjaVnz56O1ypUqMDy5cvZvn07hQoV4q233uLMmTNpDjctWrSgYsWK9OjRgylTphAbG+sUYhKucfz4cT755BPq1avHV199xYoVK5yOKV26NEeOHGHv3r2ULFmSAgUKJLkFvGvXrowZM4YePXowduxYzp07R//+/XnqqacIDg7O2F9OMl555RXGjBlDuXLlqFWrFgsXLmTv3r189NFHALz11luEhoZSu3ZtPDw8WLZsGSEhIRQsWJBFixZht9tp0KAB/v7+fPjhh/j5+REREZFl9f2duqVERMQtPfPMM1y8eJGWLVs6jY8ZOXIk9957Ly1btqR58+aEhITQvn37NJ/Xw8ODFStWcO3aNerXr8+zzz7La6+95nRM27Zteemll+jXrx+1atVi+/btjBo1yumYxx57jFatWnHfffdRrFixZG9H9/f3Z926dVy4cIF69erRqVMnHnjgAWbNmpW+v4y7GDBgAIMHD+bll1+mevXqrF27llWrVlGhQgXAvPNr8uTJ1K1bl3r16nH06FFWr16Nh4cHBQsWZP78+TRu3JgaNWqwceNGvvjiC4oUKZKlNSZmMwzDyLaz50KxsbEEBQURExNDYGCg1eWIiORJ169f58iRI5QpUwZfX1+ryxEXkdrvVXq+v9VyIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciIpJhbnZPimSzrPp9UrgREZF0S5jx9qpVK2WKS0qYoTnxUhEZoUn8REQk3Tw9PSlYsKBjfSJ/f3/HDL8iGREfH8+5c+fw9/fHyytz8UThJgvdvm2ucVKwoNWViIhkv5CQEIA0L8AocjceHh6UKlUq00FZ4SaLbNsGzz4LtWvDxx9bXY2ISPaz2WyEhoZSvHjxbF0EUdyHt7c3Hh6ZHzGjcJNF8ueHgwfht99g8GCoW9fqikREcoanp2emx0iIZCUNKM4itWrBU0+Z20OGgG4gEBERsYbCTRZ69VXw9YVvvoEvv7S6GhEREfekcJOFwsNh0CBz+1//MgcYi4iISM5SuMliw4ZB0aLm2Jv//MfqakRERNyPwk0WCwqC0aPN7TFj4PJla+sRERFxNwo32eD556F8eTh7FqZMsboaERER96Jwkw28veH1183tqVPh1Clr6xEREXEnloebd955h9KlS+Pr60uDBg3YuXNnqsdfunSJvn37Ehoaio+PDxUrVmT16tU5VG3adewIjRrB1at3uqlEREQk+1kabpYuXcrgwYMZM2YMu3fvpmbNmrRs2TLFqbxv3rzJgw8+yNGjR1m+fDkHDx5k/vz5lChRIocrvzub7U6X1MKF8Msv1tYjIiLiLmyGhevVN2jQgHr16jFr1izAXDQrPDyc/v37M2zYsCTHz507lylTpvDbb785VqRNr9jYWIKCgoiJiSEwMDBT9adFp07w2WfQujXkwgYmERGRPCE939+WtdzcvHmTH3/8kRYtWtwpxsODFi1asGPHjmTfs2rVKho2bEjfvn0JDg7mnnvuYeLEidjt9hSvc+PGDWJjY50eOWnSJPDygjVrYNOmHL20iIiIW7Is3Jw/fx673U5wcLDT/uDgYKKjo5N9zx9//MHy5cux2+2sXr2aUaNGMXXqVF599dUUrzNp0iSCgoIcj/Dw8Cz9HHdToQL06WNuv/IKxMfn6OVFRETcjuUDitMjPj6e4sWLM2/ePOrUqUNkZCQjRoxg7ty5Kb5n+PDhxMTEOB4nTpzIwYpNo0dDYCDs2QMffZTjlxcREXErloWbokWL4unpyZkzZ5z2nzlzhpCQkGTfExoaSsWKFZ1Wn61SpQrR0dHcvHkz2ff4+PgQGBjo9MhpRYvC8OHm9ogRcO1ajpcgIiLiNiwLN97e3tSpU4dNiQaixMfHs2nTJho2bJjsexo3bszhw4eJT9S38/vvvxMaGoq3t3e215wZAweaa0+dOAEzZlhdjYiIiOuytFtq8ODBzJ8/n/fff58DBw7Qp08frly5Qq9evQDo3r07wxOaPIA+ffpw4cIFBg4cyO+//85XX33FxIkT6du3r1UfIc38/MxVwwEmToTz562tR0RExFV5WXnxyMhIzp07x+jRo4mOjqZWrVqsXbvWMcj4+PHjeHjcyV/h4eGsW7eOl156iRo1alCiRAkGDhzI0KFDrfoI6dKtG7z9NuzdCxMmwPTpVlckIiLieiyd58YKOT3Pzd9t3AgPPmjeHn7ggLkGlYiIiKQuT8xz465atIBWreD27TuDjEVERCTrKNxYYPJk8PCA5cshhfkKRUREJIMUbixQvTr07GluDxkC7tUxKCIikr0Ubiwyfrx5B9X27bBihdXViIiIuA6FG4uUKGG22gAMGwa3bllbj4iIiKtQuLHQK69A8eJw6BC8+67V1YiIiLgGhRsLFSgA48aZ2+PGQUyMtfWIiIi4AoUbiz37LFSubM5Y/MYbVlcjIiKS9yncWMzL606oefttc+0pERERyTiFm1ygTRto2hSuX4dRo6yuRkREJG9TuMkFbDZ4801z+4MPzLWnREREJGMUbnKJevWgSxdzQr9XXtHEfiIiIhmlcJOLTJwI3t7m4prr1lldjYiISN6kcJOLlCkD/fqZ26+8Ana7tfWIiIjkRQo3ucyIEVCwIPzyC7z/vtXViIiI5D0KN7lM4cIwcqS5PWoUXLlibT0iIiJ5jcJNLtSvH5QuDadOmXPfiIiISNop3ORCPj7m4GIwJ/g7c8baekRERPIShZtcKjIS6taFuLg760+JiIjI3Snc5FIeHncm9ps3D377zdp6RERE8gqFm1ysWTNzaQa7HYYNs7oaERGRvEHhJpd74w3w9ITPP4etW62uRkREJPdTuMnlqlSBZ581t4cM0bIMIiIid6NwkweMHQv588POnfDpp1ZXIyIikrsp3OQBISHwr3+Z28OHw40b1tYjIiKSmync5BEvvwyhoXDkCMyebXU1IiIiuZfCTR6RPz+MH29uT5gAFy9aW4+IiEhupXCTh/TqBdWqmcEmYQZjERERcaZwk4d4esLkyeb2jBlw9GjWX8Nuhy1bYMkS80+7PeuvISIikp0UbvKY1q3h/vvh5k0YMSJrzx0VZS7Yed998OST5p+lS5v7RURE8gqFmzzGZoMpU8ztjz+GH37ImvNGRUGnTvDnn877T5409yvgiIhIXqFwkwfdey9062Zuv/JK5if2s9th4MDkz5Owb9AgdVGJiEjeoHCTR736Kvj4mONivvoqc+faujVpi01ihgEnTmj5BxERyRsUbvKoiAiztQXMCf5u3874uU6fztrjRERErKRwk4cNHw5FisCBA7BgQcbPExqatceJiIhYSeEmDytYEEaPNrdHj4a4uIydp0kTKFnSHKycHJsNwsPN40RERHI7hZs87oUXoHx5OHPmzl1U6eXpCdOnm9t/DzgJz6dNM48TERHJ7RRu8jhvb5g0ydx+8004dSpj5+nYEZYvhxIlnPeXLGnu79gxc3WKiIjkFJthZPZG4rwlNjaWoKAgYmJiCAwMtLqcLGEY0Lgx7NgBzz4L8+dn/Fx2u3lX1OnT5hibJk3UYiMiItZLz/e3wo2L2L7dDDgeHvDzz+YaVCIiIq4iPd/f6pZyEY0amV1H8fHmreEiIiLuSuHGhbz+Onh5werV8PXXVlcjIiJiDYUbF1Khgnn3FMCQIWYrjoiIiLtRuHExo0dDYCDs2WMurCkiIuJuFG5cTLFiMGyYuT1iBFy/bm09IiIiOU3hxgUNGmTOT3P8OMyYYXU1IiIiOUvhxgX5+ZmrhgNMnAh//WVtPSIiIjlJ4cZFdesGNWtCTAxMmGB1NSIiIjlH4cZFeXreWWtq9mz43/+srUdERCSnKNy4sAcfhJYt4dYtGD7c6mpERERyhsKNi5s82VzZe9ky+P57q6sRERHJfgo3Lq5GDejZ09weMsRcZFNERMSVKdy4gQkTzDuotm2DlSutrkZERCR7Kdy4gRIlYPBgc3voUHMMjoiIiKtSuHET//qXOXvxoUMwb57V1YiIiGQfhRs3ERgIY8ea2+PGQWyspeWIiIhkG4UbN9K7N1SsCOfOwRtvWF2NiIhI9lC4cSP58t0JNW+9BX/+aW09IiIi2UHhxs20awf//Ke5WvioUVZXIyIikvUUbtyMzQZvvmluv/8+/PSTtfWIiIhkNYUbN9SgAXTubE7o969/WV2NiIhI1lK4cVOTJpljcNavh3XrrK5GREQk6yjcuKmyZaFvX3P7lVfAbre2HhERkayicOPGRo6EoCDYtw8++MDqakRERLKGwo0bK1IERowwt0eOhKtXra1HREQkKyjcuLn+/SEiAk6dgrfftroaERGRzMsV4eadd96hdOnS+Pr60qBBA3bu3JnisYsWLcJmszk9fH19c7Ba1+LrCxMnmttvvAFnz1pbj4iISGZZHm6WLl3K4MGDGTNmDLt376ZmzZq0bNmSs6l8ywYGBnL69GnH49ixYzlYsevp0gXq1IHLl811p0RERPIyy8PNW2+9Re/evenVqxdVq1Zl7ty5+Pv7s2DBghTfY7PZCAkJcTyCg4NzsGLX4+FxZ2K/d9+FgwetrUdERCQzLA03N2/e5Mcff6RFixaOfR4eHrRo0YIdO3ak+L64uDgiIiIIDw+nXbt27N+/PyfKdWnNm8Ojj5q3hA8bZnU1IiIiGWdpuDl//jx2uz1Jy0twcDDR0dHJvqdSpUosWLCAzz//nA8//JD4+HgaNWrEnymsAnnjxg1iY2OdHpK8N94wW3FWroStW62uRkREJGMs75ZKr4YNG9K9e3dq1apFs2bNiIqKolixYrz77rvJHj9p0iSCgoIcj/Dw8ByuOO+oWhWefdbcfukluHnT2npEREQywtJwU7RoUTw9PTlz5ozT/jNnzhASEpKmc+TLl4/atWtz+PDhZF8fPnw4MTExjseJEycyXbcrGzcOAgPhxx+hTx9z/SkREZG8xNJw4+3tTZ06ddi0aZNjX3x8PJs2baJhw4ZpOofdbmffvn2EhoYm+7qPjw+BgYFOD0lZSAh88onZPbVgAUybZnVFIiIi6WN5t9TgwYOZP38+77//PgcOHKBPnz5cuXKFXr16AdC9e3eGDx/uOH78+PGsX7+eP/74g927d9OtWzeOHTvGswn9KZJprVvfuXtqyBBYs8baekRERNLDy+oCIiMjOXfuHKNHjyY6OppatWqxdu1axyDj48eP4+FxJ4NdvHiR3r17Ex0dTaFChahTpw7bt2+natWqVn0ElzRoEOzfD++9Z86Ds2OHOSZHREQkt7MZhnuNqoiNjSUoKIiYmBh1Ud3FzZvQooV551TZsrBzp7kelYiISE5Lz/e35d1Sknt5e8Nnn0Hp0vDHH9Cpk+6gEhGR3E/hRlJVrBh88QUEBMCWLeZCm+7V1iciInmNwo3c1T33wJIlYLPBvHkwa5bVFYmIiKRM4UbS5NFHzRmMwRxsvH69peWIiIikSOFG0mzIEOjeHeLjoXNnLbApIiK5k8KNpFlCt1SjRhATA23awIULVlclIiLiTOFG0sXHB6KioFQpOHTIbMG5dcvqqkRERO5QuJF0Cw6GVasgf37YtMlcZFNERCS3ULiRDKlZEz780Nx+5x2YM8faekRERBIo3EiGtW8PEyea2/37w9dfW1qOiIgIoHAjmTRsGHTtCna7OYPxoUNWVyQiIu5O4UYyxWaD//wH6teHixfNO6guXbK6KhERcWcKN5Jpvr6wciWULGnOfdOlC9y+bXVVIiLirhRuJEuEhsLnn4OfH6xbZ074JyIiYgWFG8ky994LH3xgbk+fDvPnW1uPiIi4J4UbyVKdOsG4ceb2iy/CN99YW4+IiLgfhRvJcqNGQWSkOe7mscfgjz+srkhERNyJwo1kOZsNFi6EunXhr7/MO6hiY62uSkRE3IXCjWQLPz/zDqrQUPj1V3jiCXMuHBERkeymcCPZpkQJ8w4qX19YvRqGDrW6IhERcQcKN5Kt6tUzu6gApk69sy0iIpJdFG4k23XpYg4yBnj+efjuO2vrERER15ahcHPixAn+/PNPx/OdO3cyaNAg5s2bl2WFiWsZO9a8c+rWLejQAY4etboiERFxVRkKN08++SSbN28GIDo6mgcffJCdO3cyYsQIxo8fn6UFimvw8ID334fateH8eWjbFi5ftroqERFxRRkKN7/88gv169cH4NNPP+Wee+5h+/btfPTRRyxatCgr6xMXkj+/OcA4OBj27YNu3SA+3uqqRETE1WQo3Ny6dQsfHx8ANm7cSNu2bQGoXLkyp0+fzrrqxOWEh5sBx8cHVq2CESOsrkhERFxNhsJNtWrVmDt3Llu3bmXDhg20atUKgFOnTlGkSJEsLVBcT4MG8N575vbrr8PixdbWIyIiriVD4eaNN97g3XffpXnz5jzxxBPUrFkTgFWrVjm6q0RS07UrDB9ubj/7LOzYYW09IiLiOmyGYRgZeaPdbic2NpZChQo59h09ehR/f3+KFy+eZQVmtdjYWIKCgoiJiSEwMNDqctxafDx07Gh2UxUvDrt2QalSVlclIiK5UXq+vzPUcnPt2jVu3LjhCDbHjh1j2rRpHDx4MFcHG8ldPDzgww+hRg04exbatYMrV6yuSkRE8roMhZt27drxwQcfAHDp0iUaNGjA1KlTad++PXPmzMnSAsW1BQSYA4uLFYO9e+Gpp3QHlYiIZE6Gws3u3btp0qQJAMuXLyc4OJhjx47xwQcfMGPGjCwtUFxfRASsWAHe3uafY8ZYXZGIiORlGQo3V69epUCBAgCsX7+ejh074uHhwT/+8Q+OHTuWpQWKe2jcGBImuH71VViyJHPns9thyxbzPFu2aEVyERF3kqFwU758eVauXMmJEydYt24dDz30EABnz57VIF3JsB494JVXzO1evWDnzoydJyoKSpeG++6DJ580/yxd2twvIiKuL0PhZvTo0QwZMoTSpUtTv359GjZsCJitOLVr187SAsW9TJoEjz4KN25A+/Zw8mT63h8VBZ06QaKlzwDzPJ06KeCIiLiDDN8KHh0dzenTp6lZsyYeHmZG2rlzJ4GBgVSuXDlLi8xKuhU894uNhUaNYP9+qFMHvv0W/P3v/j673Wyh+XuwSWCzQcmScOQIeHpmackiIpLNsv1WcICQkBBq167NqVOnHCuE169fP1cHG8kbAgPhiy+gSBH48Ufo2RPSEsG3bk052IB5jhMnzONERMR1ZSjcxMfHM378eIKCgoiIiCAiIoKCBQsyYcIE4nUfr2SBMmXMLqR8+WDZMkjLYvNpXdZMy5+JiLi2DIWbESNGMGvWLF5//XX27NnDnj17mDhxIjNnzmTUqFFZXaO4qaZNIWHapLFjzZCTmtDQtJ03rceJiEjelKExN2FhYcydO9exGniCzz//nBdffJGT6R0FmoM05ibveeklmDYN/PzMLqU6dZI/LmHMzcmTyXdjacyNiEjele1jbi5cuJDs2JrKlStz4cKFjJxSJEVTpkCrVnDtmrlEQ0rdSp6eMH26uW2zOb+W8HzaNAUbERFXl6FwU7NmTWbNmpVk/6xZs6hRo0amixJJzMsLPvkEKlc2W2XatTODTnI6doTly6FECef9JUua+zt2zP56RUTEWhnqlvrmm2945JFHKFWqlGOOmx07dnDixAlWr17tWJohN1K3VN51+DA0aAAXLsATT8BHHyVtoUlgt5tdWKdPm2NsmjRRi42ISF6W7d1SzZo14/fff6dDhw5cunSJS5cu0bFjR/bv38/ixYszVLTI3ZQvb7a+eHmZyypMnJjysZ6e0Ly5GYKaN1ewERFxJxmexC85P/30E/feey/2XLyQj1pu8r5334UXXjC3o6KgQwdr6xERkeyXI5P4iVjl+eehXz9zu1s32LvX0nJERCSXUbiRPOntt+HBB+HqVWjbFs6csboiERHJLRRuJE/y8oKlS6FiRXNJhQ4d4Pp1q6sSEZHcwCs9B3e8y320ly5dykwtIulSqJC5BlWDBrBjBzz3HLz/fsp3UImIiHtIV7gJCgq66+vdu3fPVEEi6VGxInz6KbRuDYsXwz33wL/+ZXVVIiJipSy9Wyov0N1SrmnWLOjf32y1WbnSHIcjIiKuQ3dLidvp29e8PdwwoGtX2LfP6opERMQqCjfiEmw2mDED7rsP4uKgTRs4d87qqkRExAoKN+Iy8uWDZcugXDk4dsxcR+rGDaurEhGRnKZwIy6lSBHzDqrAQPjuO+jdG27dsroqERHJSQo34nKqVDHnwPHwMO+gatwYDh2yuioREckpCjfiklq1MhfZLFgQdu2C2rVhwQJzwLGIiLg2hRtxWR06wM8/Q7NmcOUKPPMMREbCxYtWVyYiItlJ4UZcWng4bNoEEyeaSzYsWwY1asA331hdmYiIZBeFG3F5np4wfDhs3w7ly8Off5q3jP/73xpsLCLiihRuxG3Uqwd79sDTT5tjbyZN0mBjERFXpHAjbiUgAN57z1yPSoONRURck8KNuKXHH9dgYxERV6VwI24rYbDxpEkabCwi4koUbsSteXrCsGEabCwi4koUbkRIfrBxo0YabCwikhcp3Ij8v4TBxsuWmYONf/hBg41FRPKiXBFu3nnnHUqXLo2vry8NGjRg586daXrfJ598gs1mo3379tlboLiVTp3MwcbNm98ZbNy5swYbi4jkFZaHm6VLlzJ48GDGjBnD7t27qVmzJi1btuTs2bOpvu/o0aMMGTKEJk2a5FCl4k7Cw2HjxjuDjZcvNwcbb9lidWUiInI3loebt956i969e9OrVy+qVq3K3Llz8ff3Z8GCBSm+x26307VrV8aNG0fZsmVzsFpxJ4kHG1eoYA42vv9+c7ZjDTYWEcm9LA03N2/e5Mcff6RFixaOfR4eHrRo0YIdO3ak+L7x48dTvHhxnnnmmbte48aNG8TGxjo9RNKjXj3YvfvOYOPXX9dgYxGR3MzScHP+/HnsdjvBwcFO+4ODg4mOjk72Pd999x3vvfce8+fPT9M1Jk2aRFBQkOMRHh6e6brF/SQebFyokAYbi4jkZpZ3S6XH5cuXeeqpp5g/fz5FixZN03uGDx9OTEyM43HixIlsrlJcWadO8NNPSQcbX7hgdWUiIpLAy8qLFy1aFE9PT86cOeO0/8yZM4SEhCQ5/n//+x9Hjx6lTZs2jn3x8fEAeHl5cfDgQcqVK+f0Hh8fH3x8fLKhenFXCYONp0yBUaPMwcbffw+LF5uhR0RErGVpy423tzd16tRh06ZNjn3x8fFs2rSJhg0bJjm+cuXK7Nu3j7179zoebdu25b777mPv3r3qcpIco8HGIiK5l6UtNwCDBw+mR48e1K1bl/r16zNt2jSuXLlCr169AOjevTslSpRg0qRJ+Pr6cs899zi9v2DBggBJ9ovkhITBxoMGmWNyXn/dbNX5+GMz9IiISM6zPNxERkZy7tw5Ro8eTXR0NLVq1WLt2rWOQcbHjx/HwyNPDQ0SNxMQAP/5D7RuDb173xlsPGMG9OoFNpvVFYqIuBebYbjXvR6xsbEEBQURExNDYGCg1eWIizlxArp3vzPZX6dO8O67ULiwpWWJiOR56fn+VpOISBZKGGz8+ut3ZjauWVMzG4uI5CSFG5Es5ukJQ4fCjh1JBxvfvGl1dSIirk/hRiSb1K1rDjZ+5pk7Mxs3bqyZjUVEspvCjUg2ShhsvHy5ZjYWEckpCjciOeCxx+Dnn+G+++7MbPz445rZWEQkOyjciOSQkiVhw4Y7g40/++zug43tdvP1JUvMP+32HCpWRCQPU7gRyUHpGWwcFQWlS5utPU8+af5ZurS5X0REUqZwI2KBhMHGzz7rPNj499/N16OizDly/vzT+X0nT5r7FXBERFKmcCNikYAAmD8/6WDj+fNhwIDkBxwn7Bs0SF1UIiIpUbgRsVjiwcZXr8Jzz5ktNCkxDHMm5K1bc65GEZG8ROFGJBdIPNjY0zNt7zl9OntrEhHJqxRuRHKJhMHG77yTtuNDQ7O3HhGRvErhRiSXefZZCAtL+XWbzVzDqkmTnKtJRCQvUbgRyWU8PWHmTDPEJMcwYNKktHdfiYi4G4UbkVyoY0fzLqqSJZN/vV8/c26cU6dyti4RkbzAZhjutcJNbGwsQUFBxMTEEBgYaHU5Iqmy2827ok6fhoIF4X//g+nT4fBh8/V8+eCJJ+Dll6FGDUtLFRHJVun5/la4Eclj7Hb44guYOhW+++7O/gcfNEPOQw+l3KUlIpJXpef7W91SInmMpye0b2+26Hz/vbkAp4eHeSt5q1ZmC87ChXDjhtWViohYQ+FGJA9r0AA+/dTspho40Jz1+Jdf4OmnzXWoJk7UyuMi4n4UbkRcQJkyMG2aOXPxG29AiRIQHQ0jRpi3jffrZ47XERFxBwo3Ii6kYEH417/gjz9g8WKoVctc0uGdd8xVyDt2hO3bra5SRCR7KdyIuCBvb+jWzVx5fNMmaN3anB9nxQpz9fGGDc1bzbX4poi4IoUbERdms8H998Pq1bB/PzzzjBl8EgYiV6gAM2ZAXJzVlYqIZB2FGxE3UbUq/Oc/cPw4jBoFRYrAkSPmQOTwcBg2LPXVyEVE8gqFGxE3ExwM48ebIWf2bLP15tIlcyBymTLQvTv89JPVVYqIZJzCjYib8veHPn3gt99g5UpzIc5bt+4MRH7wQVi71hyrIyKSlyjciLg5Dw9o1w6+/RZ27oTISHOiwI0bzYHI1avDggWaFFBE8g6FGxFxqFcPPvnEnBRw0CBzUsCEgcilS8Nrr8Fff1ldpYhI6hRuRCSJ0qXh7bfNSQEnT74zKeDIkebg47597yzeKSKS2yjciEiKChaEV14x76r68EOoXRuuXTMHIlesCB06wLZtGpcjIrmLwo2I3FW+fNC1K/z4ozkp4MMPm4Fm5Ur45z/NSQGXLYPbt62uVERE4UZE0iFhUsCvvjLH4jz7LPj4wH//C507m7eVT58Oly9bXamIuDOFGxHJkKpVYf58OHYMRo+GokXh6FFzILImBRQRKynciEimBAfDuHHmpIBz55pjcWJizEkBS5fWpIAikvMUbkQkS/j5wfPPw4ED8Pnn0LSpOQYnYVLAFi3gyy81X46IZD+FGxHJUh4e0LYtfPMN7NoFXbqYkwJu2gRt2phrWrVta7byHDtmdbUi4opshuFeN3HGxsYSFBRETEwMgYGBVpcj4haOHTNXH//4Y3O+nMSqVjXvvmrd2rzzytvbmhpFJHdLz/e3wo2I5BjDgL17Yc0aWL0aduyA+Pg7rwcEmN1XCWGnZEnLShWRXEbhJhUKNyI5z26HrVvh9GkIDTUX6fT0hAsXYMMGM+ysWQNnzzq/r3p1M+Q8/DA0amTOtyMi7knhJhUKNyI5KyoKBg6EP/+8s69kSXM+nI4d7+yLj4c9e8wWnTVr4PvvnWc+Dgw0Vyp/+GFo1QrCwnLuM4iI9RRuUqFwI5JzoqKgU6ekyzPYbOafy5c7B5zE/voL1q83w87atXD+vPPrtWrdadX5xz/AyyvLyxeRXEThJhUKNyI5w24357lJ3GKTmM1mtuAcOWJ2Ud3tXD/+eKdVZ9cu58BUsCA89JAZdlq1gpCQrPoUIpJbKNykQuFGJGds2QL33Xf34zZvhubN03fuc+dg3Toz7KxbZ47dSaxOnTutOvXr3z08iUjul57vb81zIyLZ4vTprD0usWLFoFs389bys2dh+3YYNcoMNWC28rz6qjkIuXhxePJJczLBc+fSfy0RyXvUciMi2SI7W25SEx19p1Vn/Xq4dOnOazYb1K1751bzunXVqiOSV6hbKhUKNyI5I2HMzcmTSQcUQ/rG3GTU7dvmiuWrV5uPvXudXy9aFFq2NMNOy5bm7Mkikjsp3KRC4UYk5yTcLQXOASctd0tlh1OnzDuv1qwxW3ViY51ratDgTqvOvfeaS0mISO6gcJMKhRuRnJXcPDfh4TBtWs4Gm7+7dcucITnhDqyff3Z+vXhx886rhx8278QqVMiaOkXEpHCTCoUbkZyX0gzFucmff96ZKXnDBoiLu/Oahwc0bHjnDqxate60PolIzlC4SYXCjYjczc2bsG3bnVad/fudXw8ONruw6tUzByXXq6fxOiLZTeEmFQo3IpJex4/fWexz0ya4ciXpMWXKmCEnIfDUqQMFCuR8rSKuSuEmFQo3IpIZN27ADz+YsyQnPA4dSnqczQaVKzsHnlq1wNc3x0sWcQkKN6lQuBGRrHbpkjlxYOLAc+JE0uO8vMyVzhN3Z1WrptXORdJC4SYVCjcikhPOnEnawpPcDMm+vlC79p2wU68eVKyo29BF/k7hJhUKNyJiBcMwW3MSgs4PP5iPmJikxwYGmmN2EgeeiAjdoSXuTeEmFQo3IpJbxMfD4cPOgWf3brh2LemxRYs6d2fVq6fVz8W9KNykQuFGRHKz27fh11+dA8/PP5uTDv5dyZLOgaduXU02KK5L4SYVCjciktdcv24GnMSB59dfk1+zq3x559ade++F/PlzvmaRrKZwkwqFGxFxBXFxZhdWQtjZtQv+97+kx3l4QJUqd8JOvXpQowb4+OR8zSKZoXCTCoUbEXFVFy7cCToJf548mfS4fPnMgFO3rhl8KlUy79CKiMh9y2KIJFC4SYXCjYi4k9OnncPOrl3w11/JH+vtDeXKmUEnIfAkPIoX191aYi2Fm1Qo3IhIRuWFBUDvxjDg6FEz5OzdCwcPwu+/m7Ms37iR8vuCgpzDTkL4qVABAgJyqnpxZwo3qVC4EZGMiIqCgQPN1cMTlCwJ06dDx47W1ZVV4uPNeXgSwk7ix9GjyQ9eThAWlnxrT5kymn1Zso7CTSoUbkQkvaKioFOnpF/wCd00y5e7RsBJyfXr5mDlhLCTOAAlN+tyAi8vKFs2+Raf0FB1c0n6KNykQuFGRNLDbofSpZ1bbBKz2cwWnCNH8l4XVVa4eNHs0kquxefq1ZTflz9/8q09FSuaXWAif6dwkwqFGxFJjy1b4L777n7c5s3QvHl2V5N3GIZ5p1bisJMQgI4cMUNjSoKDk2/tKVtWt7C7s/R8f3vlUE2peuedd5gyZQrR0dHUrFmTmTNnUr9+/WSPjYqKYuLEiRw+fJhbt25RoUIFXn75ZZ566qkcrlpE3MHp01l7nLtIaNEqWRLuv9/5tZs3zYCTXGvP6dPmoqNnzpiDtxPz8DBb0f7e4lOhApQoYXaDiUAuCDdLly5l8ODBzJ07lwYNGjBt2jRatmzJwYMHKV68eJLjCxcuzIgRI6hcuTLe3t58+eWX9OrVi+LFi9OyZUsLPoGIuLLQ0Kw9TsxbzitVMh9/FxtrdnMlN77n8mX44w/zsXat8/tsNrPFJyzMDDphYXceiZ8XLaqxPu7A8m6pBg0aUK9ePWbNmgVAfHw84eHh9O/fn2HDhqXpHPfeey+PPPIIEyZMuOux6pYSkfRIGHNz8mTydwy5+5ibnGIYZmvO3wPP77+bg52TW3srOd7eZhD9e+j5+3aBAtn7eST98ky31M2bN/nxxx8ZPny4Y5+HhwctWrRgx44dd32/YRh8/fXXHDx4kDfeeCPZY27cuMGNRJM3xMbGZr5wEXEbnp7m7d6dOplBJnHASWgBmDZNwSa72WzmKughIdC0qfNr8fHmXVunTpmPkyeTbp88aR5z8yYcO2Y+UhMQcPcAFBqqMUC5laXh5vz589jtdoKDg532BwcH89tvv6X4vpiYGEqUKMGNGzfw9PRk9uzZPPjgg8keO2nSJMaNG5eldYuIe+nY0bzdO7l5bqZNc+3bwPMCDw+zSyo4GGrXTvm4mzchOjr58JN4OzbWXLsroWUoNUWKpB6AwsLM2Z0VfnOW5WNuMqJAgQLs3buXuLg4Nm3axODBgylbtizNk7lVYfjw4QwePNjxPDY2lvDw8BysVkRcQceO0K5d3p+h2J15e0OpUuYjNXFxdwJPSgHo1ClzRue//jIfP/+c8vk8Pc0Wp9QCUFgYFCqk8UBZxdJwU7RoUTw9PTlz5ozT/jNnzhASEpLi+zw8PChfvjwAtWrV4sCBA0yaNCnZcOPj44OP2g1FJAt4eup2b3cQEHDnTqyUGIa5UGly3V+J9505Y47bOnky+UVME/P1NQc8Fy5stgglPFJ7XqiQ7hJLjqV/Jd7e3tSpU4dNmzbRvn17wBxQvGnTJvr165fm88THxzuNqxEREclONtudgFG9esrH3b4NZ8/evSvswgVzJug//0x5wsiUBAWlPQwlbAcGunYrkeV5b/DgwfTo0YO6detSv359pk2bxpUrV+jVqxcA3bt3p0SJEkyaNAkwx9DUrVuXcuXKcePGDVavXs3ixYuZM2eOlR9DREQkCS+vO91Oqbl+3ezuPH/e7Oa6cOFOl1fi7cTPY2LM98bEmI8//khfXYULp6+VqEgRs3UpL7A83ERGRnLu3DlGjx5NdHQ0tWrVYu3atY5BxsePH8fDw8Nx/JUrV3jxxRf5888/8fPzo3Llynz44YdERkZa9RFEREQyxdfXXGi0TJm0v+f2bTPopBR+UgpG167daVE6ezZ9dfr5pS0IlSgB996bvnNnJcvnuclpmudGRETc2bVraQ9CibdTWzLj7+rUgR9+yNq688w8NyIiIpKz/PzuLI2RVoZh3iKf1iBUrVr21Z8WCjciIiKSKpvNHLgcFGQuYJrbedz9EBEREZG8Q+FGREREXIq6pURE3IzdrpmWxbUp3IiIuJGoqOTXyJo+XWtkietQt5SIiJuIijJXN//7DLgnT5r7o6KsqUskqynciIi4AbvdbLFJbmazhH2DBqVvLhOR3ErhRkTEDWzdmvqaRYYBJ06Yx4nkdQo3IiJu4PTprD1OJDdTuBERcQOhoVl7nEhupnAjIuIGmjQx74qy2ZJ/3WaD8HDzOJG8TuFGRMQNeHqat3tD0oCT8HzaNM13I65B4UZExE107AjLl0OJEs77S5Y092ueG3EVmsRPRMSNdOwI7dpphmJxbQo3IiJuxtMTmje3ugqR7KNuKREREXEpCjciIiLiUtQtJSIieZZWOJfkKNyIiEiepBXOJSXqlhIRkTxHK5xLahRuREQkT9EK53I3CjciIpKnaIVzuRuFGxERyVO0wrncjcKNiIjkKVrhXO5G4UZERPIUrXAud6NwIyIieYpWOJe7UbgREZE8RyucS2o0iZ+IiORJWuFcUqJwIyIieZZWOJfkKNyIiIhYTGtkZS2FGxEREQtpjayspwHFIiIiFtEaWdlD4UZERMQCWiMr+yjciIiIWEBrZGUfhRsRERELaI2s7KNwIyIiYgGtkZV9FG5EREQsoDWyso/CjYiIiAW0Rlb2UbgRERGxiNbIyh6axE9ERMRCWiMr6ynciIiIWExrZGUthRsRERHJErlljSyFGxEREcm03LRGlgYUi4iISKbktjWyFG5EREQkw3LjGlkKNyIiIpJhuXGNLIUbERERybDcuEaWwo2IiIhkWG5cI0vhRkRERDIsN66RpXAjIiIiGZYb18hSuBEREZFMyW1rZGkSPxEREcm03LRGlsKNiIiIZIncskaWuqVERETEpSjciIiIiEtRuBERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpbjdDMWGYQAQGxtrcSUiIiKSVgnf2wnf46lxu3Bz+fJlAMLDwy2uRERERNLr8uXLBAUFpXqMzUhLBHIh8fHxnDp1igIFCmD7+9rsApjpODw8nBMnThAYGGh1OW5PP4/cRT+P3Ec/k9wlu34ehmFw+fJlwsLC8PBIfVSN27XceHh4ULJkSavLyBMCAwP1P4pcRD+P3EU/j9xHP5PcJTt+HndrsUmgAcUiIiLiUhRuRERExKUo3EgSPj4+jBkzBh8fH6tLEfTzyG3088h99DPJXXLDz8PtBhSLiIiIa1PLjYiIiLgUhRsRERFxKQo3IiIi4lIUbkRERMSlKNyIw6RJk6hXrx4FChSgePHitG/fnoMHD1pdlgCvv/46NpuNQYMGWV2KWzt58iTdunWjSJEi+Pn5Ub16dX744Qery3JLdrudUaNGUaZMGfz8/ChXrhwTJkxI07pDknnffvstbdq0ISwsDJvNxsqVK51eNwyD0aNHExoaip+fHy1atODQoUM5Vp/CjTh888039O3bl++//54NGzZw69YtHnroIa5cuWJ1aW5t165dvPvuu9SoUcPqUtzaxYsXady4Mfny5WPNmjX8+uuvTJ06lUKFClldmlt64403mDNnDrNmzeLAgQO88cYbTJ48mZkzZ1pdmlu4cuUKNWvW5J133kn29cmTJzNjxgzmzp3Lf//7X/Lnz0/Lli25fv16jtSnW8ElRefOnaN48eJ88803NG3a1Opy3FJcXBz33nsvs2fP5tVXX6VWrVpMmzbN6rLc0rBhw9i2bRtbt261uhQBHn30UYKDg3nvvfcc+x577DH8/Pz48MMPLazM/dhsNlasWEH79u0Bs9UmLCyMl19+mSFDhgAQExNDcHAwixYtokuXLtlek1puJEUxMTEAFC5c2OJK3Fffvn155JFHaNGihdWluL1Vq1ZRt25dHn/8cYoXL07t2rWZP3++1WW5rUaNGrFp0yZ+//13AH766Se+++47WrdubXFlcuTIEaKjo53+vxUUFESDBg3YsWNHjtTgdgtnStrEx8czaNAgGjduzD333GN1OW7pk08+Yffu3ezatcvqUgT4448/mDNnDoMHD+bf//43u3btYsCAAXh7e9OjRw+ry3M7w4YNIzY2lsqVK+Pp6Yndbue1116ja9euVpfm9qKjowEIDg522h8cHOx4Lbsp3Eiy+vbtyy+//MJ3331ndSlu6cSJEwwcOJANGzbg6+trdTmCGfjr1q3LxIkTAahduza//PILc+fOVbixwKeffspHH33Exx9/TLVq1di7dy+DBg0iLCxMPw9Rt5Qk1a9fP7788ks2b95MyZIlrS7HLf3444+cPXuWe++9Fy8vL7y8vPjmm2+YMWMGXl5e2O12q0t0O6GhoVStWtVpX5UqVTh+/LhFFbm3V155hWHDhtGlSxeqV6/OU089xUsvvcSkSZOsLs3thYSEAHDmzBmn/WfOnHG8lt0UbsTBMAz69evHihUr+PrrrylTpozVJbmtBx54gH379rF3717Ho27dunTt2pW9e/fi6elpdYlup3HjxkmmRvj999+JiIiwqCL3dvXqVTw8nL/CPD09iY+Pt6giSVCmTBlCQkLYtGmTY19sbCz//e9/adiwYY7UoG4pcejbty8ff/wxn3/+OQUKFHD0jQYFBeHn52dxde6lQIECScY65c+fnyJFimgMlEVeeuklGjVqxMSJE+ncuTM7d+5k3rx5zJs3z+rS3FKbNm147bXXKFWqFNWqVWPPnj289dZbPP3001aX5hbi4uI4fPiw4/mRI0fYu3cvhQsXplSpUgwaNIhXX32VChUqUKZMGUaNGkVYWJjjjqpsZ4j8PyDZx8KFC60uTQzDaNasmTFw4ECry3BrX3zxhXHPPfcYPj4+RuXKlY158+ZZXZLbio2NNQYOHGiUKlXK8PX1NcqWLWuMGDHCuHHjhtWluYXNmzcn+33Ro0cPwzAMIz4+3hg1apQRHBxs+Pj4GA888IBx8ODBHKtP89yIiIiIS9GYGxEREXEpCjciIiLiUhRuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiLglm83GypUrrS5DRLKBwo2I5LiePXtis9mSPFq1amV1aSLiArS2lIhYolWrVixcuNBpn4+Pj0XViIgrUcuNiFjCx8eHkJAQp0ehQoUAs8tozpw5tG7dGj8/P8qWLcvy5cud3r9v3z7uv/9+/Pz8KFKkCM899xxxcXFOxyxYsIBq1arh4+NDaGgo/fr1c3r9/PnzdOjQAX9/fypUqMCqVascr128eJGuXbtSrFgx/Pz8qFChQpIwJiK5k8KNiORKo0aN4rHHHuOnn36ia9eudOnShQMHDgBw5coVWrZsSaFChdi1axfLli1j48aNTuFlzpw59O3bl+eee459+/axatUqypcv73SNcePG0blzZ37++WcefvhhunbtyoULFxzX//XXX1mzZg0HDhxgzpw5FC1aNOf+AkQk43JsiU4Rkf/Xo0cPw9PT08ifP7/T47XXXjMMw1yh/oUXXnB6T4MGDYw+ffoYhmEY8+bNMwoVKmTExcU5Xv/qq68MDw8PIzo62jAMwwgLCzNGjBiRYg2AMXLkSMfzuLg4AzDWrFljGIZhtGnTxujVq1fWfGARyVEacyMilrjvvvuYM2eO077ChQs7ths2bOj0WsOGDdm7dy8ABw4coGbNmuTPn9/xeuPGjYmPj+fgwYPYbDZOnTrFAw88kGoNNWrUcGznz5+fwMBAzp49C0CfPn147LHH2L17Nw899BDt27enUaNGGfqsIpKzFG5ExBL58+dP0k2UVfz8/NJ0XL58+Zye22w24uPjAWjdujXHjh1j9erVbNiwgQceeIC+ffvy5ptvZnm9IpK1NOZGRHKl77//PsnzKlWqAFClShV++uknrly54nh927ZteHh4UKlSJQoUKEDp0qXZtGlTpmooVqwYPXr04MMPP2TatGnMmzcvU+cTkZyhlhsRscSNGzeIjo522ufl5eUYtLts2TLq1q3LP//5Tz766CN27tzJe++9B0DXrl0ZM2YMPXr0YOzYsZw7d47+/fvz1FNPERwcDMDYsWN54YUXKF68OK1bt+by5cts27aN/v37p6m+0aNHU6dOHapVq8aNGzf48ssvHeFKRHI3hRsRscTatWsJDQ112lepUiV+++03wLyT6ZNPPuHFF18kNDSUJUuWULVqVQD8/f1Zt24dAwcOpF69evj7+/PYY4/x1ltvOc7Vo0cPrl+/zttvv82QIUMoWrQonTp1SnN93t7eDB8+nKNHj+Ln50eTJk345JNPsuCTi0h2sxmGYVhdhIhIYjabjRUrVtC+fXurSxGRPEhjbkRERMSlKNyIiIiIS9GYGxHJddRbLiKZoZYbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSn/ByuJn8U9bHqcAAAAAElFTkSuQmCC", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "acc = history_dict['binary_accuracy']\n", - "val_acc = history_dict['val_binary_accuracy']\n", - "loss = history_dict['loss']\n", - "val_loss = history_dict['val_loss']\n", - "\n", - "epochs = range(1, len(acc) + 1)\n", - "\n", - "# \"bo\" is for \"blue dot\"\n", - "plt.plot(epochs, loss, 'bo', label='Training loss')\n", - "# b is for \"solid blue line\"\n", - "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n", - "plt.title('Training and validation loss')\n", - "plt.xlabel('Epochs')\n", - "plt.ylabel('Loss')\n", - "plt.legend()\n", - "\n", - "plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 28, - "id": "af51178e-fe0b-40ca-9260-2190fb52d960", - "metadata": {}, - "outputs": [ - { - "data": { - "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHHCAYAAABXx+fLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABXZUlEQVR4nO3deXhMZ/8G8HsySSb7IoksEgmhtsZObEErbdCmiCWWElu1ilLVonZetKjGVsobay0pQr1VKlJqX4pYaqm1ISTEkkgQyeT8/ji/TDIyiZlsZ5Jzf65rLplnzjnznckwt+d5znMUgiAIICIiIpIRE6kLICIiIiptDEBEREQkOwxAREREJDsMQERERCQ7DEBEREQkOwxAREREJDsMQERERCQ7DEBEREQkOwxAREREJDsMQETFoH///vDx8SnUvlOnToVCoSjegozMrVu3oFAosHr16lJ93v3790OhUGD//v2aNn1/VyVVs4+PD/r371+sxyQiwzEAUbmmUCj0uuX+giQqqiNHjmDq1Kl48uSJ1KUQUT5MpS6AqCStW7dO6/7atWsRHR2dp71WrVpFep4VK1YgKyurUPtOnDgR48aNK9Lzk/6K8rvS15EjRzBt2jT0798fDg4OWo9duXIFJib8vyeR1BiAqFz78MMPte4fO3YM0dHRedpf9ezZM1hZWen9PGZmZoWqDwBMTU1hasq/iqWlKL+r4qBSqSR9/rIiLS0N1tbWUpdB5Rj/G0Ky17ZtW7z55ps4deoUWrduDSsrK3z99dcAgF9++QXvvfcePDw8oFKp4OvrixkzZkCtVmsd49V5JdnzR+bNm4fly5fD19cXKpUKTZo0wcmTJ7X21TUHSKFQYPjw4di+fTvefPNNqFQq1KlTB7t3785T//79+9G4cWNYWFjA19cXP/74o97zig4ePIju3bujcuXKUKlU8PLywueff47nz5/neX02NjaIj49H586dYWNjAxcXF4wZMybPe/HkyRP0798f9vb2cHBwQFhYmF5DQX/99RcUCgXWrFmT57Hff/8dCoUCv/76KwDg33//xaeffooaNWrA0tISTk5O6N69O27duvXa59E1B0jfms+dO4f+/fujatWqsLCwgJubGwYOHIiHDx9qtpk6dSq+/PJLAECVKlU0w6zZtemaA3Tjxg10794dFSpUgJWVFZo1a4adO3dqbZM9n+nnn3/GzJkz4enpCQsLC7Rr1w7Xrl177es25D178uQJPv/8c/j4+EClUsHT0xP9+vVDUlKSZpsXL15g6tSpeOONN2BhYQF3d3eEhITg+vXrWvW+Orysa25V9ufr+vXr6NixI2xtbdGnTx8A+n9GAeDy5cvo0aMHXFxcYGlpiRo1amDChAkAgH379kGhUGDbtm159tuwYQMUCgWOHj362veRyg/+t5MIwMOHD9GhQwf07NkTH374IVxdXQEAq1evho2NDUaPHg0bGxv88ccfmDx5MlJSUjB37tzXHnfDhg14+vQpPv74YygUCsyZMwchISG4cePGa3siDh06hKioKHz66aewtbXFwoUL0bVrV8TFxcHJyQkAcObMGbRv3x7u7u6YNm0a1Go1pk+fDhcXF71e9+bNm/Hs2TMMHToUTk5OOHHiBBYtWoQ7d+5g8+bNWtuq1WoEBQXB398f8+bNw969e/Hdd9/B19cXQ4cOBQAIgoBOnTrh0KFD+OSTT1CrVi1s27YNYWFhr62lcePGqFq1Kn7++ec820dGRsLR0RFBQUEAgJMnT+LIkSPo2bMnPD09cevWLSxduhRt27bFxYsXDeq9M6Tm6Oho3LhxAwMGDICbmxv+/vtvLF++HH///TeOHTsGhUKBkJAQ/PPPP9i4cSO+//57ODs7A0C+v5PExES0aNECz549w2effQYnJyesWbMGH3zwAbZs2YIuXbpobf/NN9/AxMQEY8aMQXJyMubMmYM+ffrg+PHjBb5Ofd+z1NRUBAQE4NKlSxg4cCAaNmyIpKQk7NixA3fu3IGzszPUajXef/99xMTEoGfPnhg5ciSePn2K6OhoXLhwAb6+vnq//9kyMzMRFBSEVq1aYd68eZp69P2Mnjt3DgEBATAzM8OQIUPg4+OD69ev43//+x9mzpyJtm3bwsvLC+vXr8/znq5fvx6+vr5o3ry5wXVTGSYQyciwYcOEVz/2bdq0EQAIy5Yty7P9s2fP8rR9/PHHgpWVlfDixQtNW1hYmODt7a25f/PmTQGA4OTkJDx69EjT/ssvvwgAhP/973+atilTpuSpCYBgbm4uXLt2TdN29uxZAYCwaNEiTVtwcLBgZWUlxMfHa9quXr0qmJqa5jmmLrpe3+zZswWFQiH8+++/Wq8PgDB9+nStbRs0aCA0atRIc3/79u0CAGHOnDmatszMTCEgIEAAIKxatarAesaPHy+YmZlpvWfp6emCg4ODMHDgwALrPnr0qABAWLt2raZt3759AgBh3759Wq8l9+/KkJp1Pe/GjRsFAMKBAwc0bXPnzhUACDdv3syzvbe3txAWFqa5P2rUKAGAcPDgQU3b06dPhSpVqgg+Pj6CWq3Wei21atUS0tPTNdsuWLBAACCcP38+z3Plpu97NnnyZAGAEBUVlWf7rKwsQRAEYeXKlQIAYf78+fluo+u9F4Scvxu539fsz9e4ceP0qlvXZ7R169aCra2tVlvuegRB/HypVCrhyZMnmrb79+8LpqamwpQpU/I8D5VvHAIjgjgvY8CAAXnaLS0tNT8/ffoUSUlJCAgIwLNnz3D58uXXHjc0NBSOjo6a+wEBAQDEIY/XCQwM1PqfdN26dWFnZ6fZV61WY+/evejcuTM8PDw021WrVg0dOnR47fEB7deXlpaGpKQktGjRAoIg4MyZM3m2/+STT7TuBwQEaL2W3377DaamppoeIQBQKpUYMWKEXvWEhoYiIyMDUVFRmrY9e/bgyZMnCA0N1Vl3RkYGHj58iGrVqsHBwQGnT5/W67kKU3Pu533x4gWSkpLQrFkzADD4eXM/f9OmTdGqVStNm42NDYYMGYJbt27h4sWLWtsPGDAA5ubmmvv6fqb0fc+2bt2KevXq5eklAaAZVt26dSucnZ11vkdFWdIh9+9AV935fUYfPHiAAwcOYODAgahcuXK+9fTr1w/p6enYsmWLpi0yMhKZmZmvnRdI5Q8DEBGASpUqaX2pZPv777/RpUsX2Nvbw87ODi4uLpp/KJOTk1973Ff/Mc4OQ48fPzZ43+z9s/e9f/8+nj9/jmrVquXZTlebLnFxcejfvz8qVKigmdfTpk0bAHlfn4WFRZ5hnNz1AOI8E3d3d9jY2GhtV6NGDb3qqVevHmrWrInIyEhNW2RkJJydnfH2229r2p4/f47JkyfDy8sLKpUKzs7OcHFxwZMnT/T6veRmSM2PHj3CyJEj4erqCktLS7i4uKBKlSoA9Ps85Pf8up4r+8zEf//9V6u9sJ8pfd+z69ev48033yzwWNevX0eNGjWKdfK+qakpPD0987Tr8xnNDn+vq7tmzZpo0qQJ1q9fr2lbv349mjVrpvffGSo/OAeICNr/y8z25MkTtGnTBnZ2dpg+fTp8fX1hYWGB06dPY+zYsXqdSq1UKnW2C4JQovvqQ61W45133sGjR48wduxY1KxZE9bW1oiPj0f//v3zvL786iluoaGhmDlzJpKSkmBra4sdO3agV69eWl+2I0aMwKpVqzBq1Cg0b94c9vb2UCgU6NmzZ4me4t6jRw8cOXIEX375JerXrw8bGxtkZWWhffv2JX5qfbbCfi5K+z3Lryfo1Unz2VQqVZ7lAQz9jOqjX79+GDlyJO7cuYP09HQcO3YMixcvNvg4VPYxABHlY//+/Xj48CGioqLQunVrTfvNmzclrCpHxYoVYWFhofMMIH3OCjp//jz++ecfrFmzBv369dO0R0dHF7omb29vxMTEIDU1VatH5cqVK3ofIzQ0FNOmTcPWrVvh6uqKlJQU9OzZU2ubLVu2ICwsDN99952m7cWLF4VaeFDfmh8/foyYmBhMmzYNkydP1rRfvXo1zzENGQby9vbW+f5kD7F6e3vrfayC6Pue+fr64sKFCwUey9fXF8ePH0dGRka+k/mze6ZePf6rPVoF0fczWrVqVQB4bd0A0LNnT4wePRobN27E8+fPYWZmpjW8SvLBITCifGT/Tzv3/6xfvnyJH374QaqStCiVSgQGBmL79u24e/eupv3atWvYtWuXXvsD2q9PEAQsWLCg0DV17NgRmZmZWLp0qaZNrVZj0aJFeh+jVq1a8PPzQ2RkJCIjI+Hu7q4VQLNrf7XHY9GiRfn2LhRHzbreLwAIDw/Pc8zs9Wv0CWQdO3bEiRMntE7BTktLw/Lly+Hj44PatWvr+1IKpO971rVrV5w9e1bn6eLZ+3ft2hVJSUk6e06yt/H29oZSqcSBAwe0Hjfk74++n1EXFxe0bt0aK1euRFxcnM56sjk7O6NDhw746aefsH79erRv315zph7JC3uAiPLRokULODo6IiwsDJ999hkUCgXWrVtXbENQxWHq1KnYs2cPWrZsiaFDh0KtVmPx4sV48803ERsbW+C+NWvWhK+vL8aMGYP4+HjY2dlh69ates1Pyk9wcDBatmyJcePG4datW6hduzaioqIMnh8TGhqKyZMnw8LCAoMGDcozNPL+++9j3bp1sLe3R+3atXH06FHs3btXszxASdRsZ2eH1q1bY86cOcjIyEClSpWwZ88enT2CjRo1AgBMmDABPXv2hJmZGYKDg3Uu7Ddu3Dhs3LgRHTp0wGeffYYKFSpgzZo1uHnzJrZu3Vpsq0br+559+eWX2LJlC7p3746BAweiUaNGePToEXbs2IFly5ahXr166NevH9auXYvRo0fjxIkTCAgIQFpaGvbu3YtPP/0UnTp1gr29Pbp3745FixZBoVDA19cXv/76K+7fv693zYZ8RhcuXIhWrVqhYcOGGDJkCKpUqYJbt25h586def4u9OvXD926dQMAzJgxw/A3k8qHUj/vjEhC+Z0GX6dOHZ3bHz58WGjWrJlgaWkpeHh4CF999ZXw+++/v/bU6uxTfefOnZvnmAC0TrnN7zT4YcOG5dn31VOoBUEQYmJihAYNGgjm5uaCr6+v8N///lf44osvBAsLi3zehRwXL14UAgMDBRsbG8HZ2Vn46KOPNKfbv3qasrW1dZ79ddX+8OFDoW/fvoKdnZ1gb28v9O3bVzhz5oxep8Fnu3r1qgBAACAcOnQoz+OPHz8WBgwYIDg7Ows2NjZCUFCQcPny5Tzvjz6nwRtS8507d4QuXboIDg4Ogr29vdC9e3fh7t27eX6ngiAIM2bMECpVqiSYmJhonRKv63d4/fp1oVu3boKDg4NgYWEhNG3aVPj111+1tsl+LZs3b9Zq13VauS76vmfZ78fw4cOFSpUqCebm5oKnp6cQFhYmJCUlabZ59uyZMGHCBKFKlSqCmZmZ4ObmJnTr1k24fv26ZpsHDx4IXbt2FaysrARHR0fh448/Fi5cuKD350sQ9P+MCoIgXLhwQfP7sbCwEGrUqCFMmjQpzzHT09MFR0dHwd7eXnj+/HmB7xuVXwpBMKL/zhJRsejcuTP+/vtvnfNTiOQuMzMTHh4eCA4ORkREhNTlkEQ4B4iojHv1kgBXr17Fb7/9hrZt20pTEJGR2759Ox48eKA1sZrkhz1ARGWcu7u75vpU//77L5YuXYr09HScOXMG1atXl7o8IqNx/PhxnDt3DjNmzICzs3OhF6+k8oGToInKuPbt22Pjxo1ISEiASqVC8+bNMWvWLIYfolcsXboUP/30E+rXr691MVaSJ/YAERERkexwDhARERHJDgMQERERyQ7nAOmQlZWFu3fvwtbWtkhXNiYiIqLSIwgCnj59Cg8Pj9cuIsoApMPdu3fh5eUldRlERERUCLdv34anp2eB2zAA6WBrawtAfAPt7OwkroaIiIj0kZKSAi8vL833eEEYgHTIHvays7NjACIiIipj9Jm+wknQREREJDsMQERERCQ7DEBEREQkOwxAREREJDsMQERERCQ7DEBEREQkOwxAREREJDsMQERERCQ7DEBEREQkO1wJmoiIiEqFWg0cPAjcuwe4uwMBAYBSKU0tDEBERERU4qKigJEjgTt3cto8PYEFC4CQkNKvh0NgREREVKKiooBu3bTDDwDEx4vtUVGlXxMDEBEREZUYtVrs+RGEvI9lt40aJW5XmhiAiIiIqMQcPJi35yc3QQBu3xa3K00MQERERFRi7t0r3u2KCwMQERERlRh39+LdrrgwABEREVGJCQgQz/ZSKHQ/rlAAXl7idqWJAYiIiIhKjFIpnuoO5A1B2ffDw0t/PSAGICIiIiOnVgP79wMbN4p/lvYZU0UVEgJs2QJUqqTd7ukptkuxDhAXQiQiIjJixraAYGGFhACdOhnPStAKQdB1Zr68paSkwN7eHsnJybCzs5O6HCIikqnsBQRf/abOHjqSqvfEWBny/c0hMCIiIiNkrAsIlhcMQEREREbIWBcQLC8YgIiIiIyQsS4gWF4wABERERkhY11AsLxgACIiIjJCxrqAYHnBAEREROVWWV4/x1gXECwvGICIiKhciooCfHyAt94CevcW//TxEdvLCmNcQLC84DpAOnAdICKisq28rZ+jVhvPAoLGzJDvbwYgHRiAiIjKLrVa7OnJ7xRyhULsQbl5kyGivOFCiEREJFtcP4f0wQBERETlCtfPIX0wABERUbnC9XNIHwxARERUrnD9HNIHAxAREZUrXD+H9CF5AFqyZAl8fHxgYWEBf39/nDhxIt9tMzIyMH36dPj6+sLCwgL16tXD7t27i3RMIiIqf7h+Dr2OpAEoMjISo0ePxpQpU3D69GnUq1cPQUFBuH//vs7tJ06ciB9//BGLFi3CxYsX8cknn6BLly44c+ZMoY9JRETlU0gIcOsWsG8fsGGD+OfNmww/JJJ0HSB/f380adIEixcvBgBkZWXBy8sLI0aMwLhx4/Js7+HhgQkTJmDYsGGatq5du8LS0hI//fRToY6pC9cBIiIiKnvKxDpAL1++xKlTpxAYGJhTjIkJAgMDcfToUZ37pKenw8LCQqvN0tIShw4dKvQxiYgor7J8DS0ifUgWgJKSkqBWq+Hq6qrV7urqioSEBJ37BAUFYf78+bh69SqysrIQHR2NqKgo3Pv/xRwKc0xADFYpKSlaNyIiuSoP19Aieh3JJ0EbYsGCBahevTpq1qwJc3NzDB8+HAMGDICJSdFexuzZs2Fvb6+5eXl5FVPFRERlS/Y1tF5dSTk+XmxnCKLyQrIA5OzsDKVSicTERK32xMREuLm56dzHxcUF27dvR1paGv79919cvnwZNjY2qFq1aqGPCQDjx49HcnKy5nb79u0ivjoiorJHrQZGjsx7AVEgp23UKA6HUfkgWQAyNzdHo0aNEBMTo2nLyspCTEwMmjdvXuC+FhYWqFSpEjIzM7F161Z06tSpSMdUqVSws7PTuhERyQ2voUVyYirlk48ePRphYWFo3LgxmjZtivDwcKSlpWHAgAEAgH79+qFSpUqYPXs2AOD48eOIj49H/fr1ER8fj6lTpyIrKwtfffWV3sckIiLdeA0tkhNJA1BoaCgePHiAyZMnIyEhAfXr18fu3bs1k5jj4uK05ve8ePECEydOxI0bN2BjY4OOHTti3bp1cHBw0PuYRESkG6+hRXIi6TpAxorrABGRHKnV4tle8fG65wEpFOJKyjdv8jISZJzKxDpARERkXHgNLZITBiAiItLgNbRILiSdA0RERMYnJATo1Ek82+vePXHOT0AAe36ofGEAIiKiPJRKoG1bqasgKjkcAiMiIiLZYQAiIiIi2WEAIiIiItnhHCAiomKkVnPyMFFZwABERFRMoqLEi4nmvp6Wp6e4tg5PHycyLhwCIyIqBlFRQLdueS8mGh8vtkdFSVMXEenGAEREVERqtdjzo+vyEdlto0aJ2xGRcWAAIiIqooMH8/b85CYIwO3b4nZEZBwYgIiIiujeveLdjohKHgMQEVERubsX73ZEVPIYgIiIiiggQDzb69UrqGdTKAAvL3E7IjIODEBEREWkVIqnugN5Q1D2/fBwrgdEZEwYgIiIikFICLBlC1Cpkna7p6fYznWAiIwLF0IkIiomISFAp05cCZqoLGAAIiIqRkol0Lat1FUQ0etwCIyIiIhkhwGIiIiIZIdDYERkFHgVdSIqTQxARCQ5XkWdiEobAxARSSr7KuqvXkg0+yrqPIWcqOzKyABSU8Xb06faP1erBtSpI11tCkHQdf1ieUtJSYG9vT2Sk5NhZ2cndTlE5ZZaDfj45H8hUYVC7Am6eZPDYUQlLSsLSEvLCSqvBpbC/Jyenv/zTZwIzJhRvK/BkO9v9gARkWQMuYo6Ty0nyiEIwPPn+oUQfQPLs2clV6+5OWBjI95sbcU/pb42HgMQEUmGV1EnOXv+HHj82LDbkyc5oSUrq2TqMjHRDirZfxr6c+42c/OSqbUoGICISDK8ijqVZdm9MIaGmOxbQcNDhrC2Llo4efVnC4v8L+xbnjAAEZFksq+iHh+fdxI0kDMHiFdRp5IiCOLQT2FDzMuXRXt+ExPAwQFwdNTv5uAghpTs0GJtLR6DDMcARESSyb6KerduYtjJHYJ4FXXSlyCIk3cLG2IyMor2/EqlYSEm983WlgFGKgxARCSp7Kuo61oHKDycp8DLhSCI81oKG2IyM4v2/KamOT0shQkxchgyKm8YgIhIcryKevkgCEBKSsETeAt6rKghxsyscL0wjo7iUBJDjLwwABGRUeBV1I2HWg0kJQGJicCDB4adoVTUM5PMzQsfYqysGGJIfwxAREQyoFaLYSYxUbwlJOT8nPuWkCCGn6IEGZWq8CHG0pIhhkoHAxARURmVmZkTagoKNImJYqgxZN1/hQJwcgIqVixciCEydgxARERGJCNDDDWvCzSJicDDh4aHGhcXwNVV++bmlrfNxUWcGExUXvHjTURUwl6+BO7ff32gyQ41hjAxyRtqdAUaV1fA2Zmhhigb/yoQERVCerruUKOr5+bRI8OOrVTmhJr8wkz2Y05OPFuOqDAYgIiI/l96+uvDTHb7kyeGHVupFOfT5Bdocrc7OXFxPKKSxgBEROXaixf6TRJOTASSkw07tqnp68NM9q1CBYYaImPCAEREZc6zZ/rNp0lMFBfmM4SZmX6BxtVVPOOJoYaobGIAIirj1Orys4Ly/fvA9esF99gkJgJPnxp2XHNz/QKNm5t4KQSuQ0NU/jEAEZVhUVG6r6G1YIHxX0MrKwu4eBE4cgQ4fFi8Xb+u//4qlX6BxtUVsLdnqCEibQxARGVUVJR4FfVX14GJjxfbt2wxrhCUlgacOCEGnSNHgKNH804kViiAypX1myhsZ8dQQ0SFpxAEQ5bRkoeUlBTY29sjOTkZdnZ2UpdDlIdaDfj4aPf85KZQiD1BN29KNxx25452705srFh3btbWgL8/0KIF0LIl0KyZOARFRFQYhnx/sweIqAw6eDD/8AOIvUK3b4vblcYFRjMzgfPnc3p3Dh8G4uLybufpKQadli3F0FOvHhfmIyJp8J8eojLo3r3i3c5QycnA8eM5vTvHjwOpqdrbmJgA9evn9O60bAl4eZVMPUREhmIAIiqD3N2Ld7uCCAJw65b2cNb583nnHtnZAc2b5/Tu+PsDNjZFf34iopLAAERUBgUEiMNJ8fG6L4aZPQcoIMDwY2dkAGfOaA9n6epJqlpVu3endu2ye/o9EckPAxBRGaRUiqe6d+smhp3cISj7zKjwcP0CyaNH4hlZ2b07J08Cz59rb2NmBjRsmNO706JF8fQuERFJhQGIqIwKCRFPdde1DlB4uO5T4AUBuHpVu3fn0qW821WooN2707gxYGlZYi+FiKjUMQARlWEhIUCnTvmvBP3iBXDqVE7vzpEjQFJS3uPUqKEdeN54g5d4IKLyjQGIqIxTKnNOdb9/H9ixI6d359Qp4OVL7e1VKqBJE+3hLGfnUi+biEhSDEBEZZggAMeOAatXA3/8AVy7lnebihVzenZatgQaNBBDEBGRnDEAEZVBDx8C69YB//0v8PffOe0KBVCnTk7vTsuW4tlavGQEEZE2BiCiMiIrC9i3Tww9UVE5Q1uWlkBoKNC9uxh6eCkJIqLXYwAiMnJ374pDXBERwI0bOe2NGgGDBwO9eolXOyciIv0xABEZocxMYNcusbdn586ci4ja2wN9+ojBp0EDaWskIirLGICIjMiNG8DKlcCqVWLPT7aAADH0dOsGWFlJVx8RUXnBAEQksfR0YPt2YMUKICYmp93FBQgLAwYNAmrWlKw8IqJyiQGISCJ//y0Oca1bJ57VBYhna737rtjb88EHgLm5tDUSEZVXDEBEpSgtDYiMFIPP0aM57Z6ewMCB4s3bW7r6iIjkggGIqIQJgrgi84oVwMaNwNOnYrupKRAcLPb2BAXxSupERKWJAYiohDx+DKxfL/b2nD2b016tmhh6wsIANzfp6iMikjMGIKJiJAjihUlXrBCv1P7ihdiuUolncA0eDLRpw5WZiYikJvn1npcsWQIfHx9YWFjA398fJ06cKHD78PBw1KhRA5aWlvDy8sLnn3+OF9nfMgCmTp0KhUKhdavJU2iohCUmAnPmiGdrtWkD/PSTGH78/ICFC8Urtf/0k3jRUoYfIiLpSdoDFBkZidGjR2PZsmXw9/dHeHg4goKCcOXKFVSsWDHP9hs2bMC4ceOwcuVKtGjRAv/88w/69+8PhUKB+fPna7arU6cO9u7dq7lvasqOLip+ajUQHS329uzYIS5eCAA2NuLqzB99BDRuzMBDRGSMJE0G8+fPx0cffYQBAwYAAJYtW4adO3di5cqVGDduXJ7tjxw5gpYtW6J3794AAB8fH/Tq1QvHjx/X2s7U1BRunFxBJSQuTlyscOVK4PbtnPZmzcTQ06OHGIKIiMh4STYE9vLlS5w6dQqBgYE5xZiYIDAwEEdznx+cS4sWLXDq1CnNMNmNGzfw22+/oWPHjlrbXb16FR4eHqhatSr69OmDuLi4AmtJT09HSkqK1o0ot5cvga1bgQ4dAB8fYNo0MfxUqACMHAmcPy+e1j5wIMMPEVFZIFkPUFJSEtRqNVxdXbXaXV1dcfnyZZ379O7dG0lJSWjVqhUEQUBmZiY++eQTfP3115pt/P39sXr1atSoUQP37t3DtGnTEBAQgAsXLsDW1lbncWfPno1p06YV34ujcuPKFfEipGvWAPfv57S//bbY29O5M2BhIVl5RERUSJJPgjbE/v37MWvWLPzwww84ffo0oqKisHPnTsyYMUOzTYcOHdC9e3fUrVsXQUFB+O233/DkyRP8/PPP+R53/PjxSE5O1txu5x7XINl5/lxcnblNG3FS89y5YvhxcwPGjweuXRMvWdGzJ8MPEVFZJVkPkLOzM5RKJRITE7XaExMT852/M2nSJPTt2xeDBw8GAPj5+SEtLQ1DhgzBhAkTYGKSN885ODjgjTfewLVr1/KtRaVSQaVSFeHVUHkQGyuu2fPTT0BysthmYgJ07Cj29nTsKC5eSEREZZ9kPUDm5uZo1KgRYnJd/TErKwsxMTFo3ry5zn2ePXuWJ+Qo/3/5XEEQdO6TmpqK69evw93dvZgqp/IkJQX48UegSROgQQNgyRIx/Pj4ADNmiBOe//c/8bpcDD9EROWHpP+kjx49GmFhYWjcuDGaNm2K8PBwpKWlac4K69evHypVqoTZs2cDAIKDgzF//nw0aNAA/v7+uHbtGiZNmoTg4GBNEBozZgyCg4Ph7e2Nu3fvYsqUKVAqlejVq5dkr5OMjyCI83qGDxevzwUAZmZAly5ib8/bb4u9P0REVD5JGoBCQ0Px4MEDTJ48GQkJCahfvz52796tmRgdFxen1eMzceJEKBQKTJw4EfHx8XBxcUFwcDBmzpyp2ebOnTvo1asXHj58CBcXF7Rq1QrHjh2Di4tLqb8+Mk6PH4vX4Dp8WLvd2RkIDQVynZhIRETllELIb+xIxlJSUmBvb4/k5GTY2dlJXQ4Voz//FC9JkZSU97HsBQu3bAFCQkq3LiIiKjpDvr/ZyU+ykJEBTJgAvPWW7vADiMNiADBqlLjKMxERlV8MQFTuXb0KtGgBzJqVE3LyIwjiAocHD5ZObUREJA0GICq3BEFcxLB+feCvvwBHR7F3Rx/37pVkZUREJDUGICqXHj4U5/oMHgw8eyae1XXuHNCpk377c9UEIqLyjQGIyp2YGKBuXSAqSjy1fc4c8artnp5AQID4Z35XaFcoAC8vcTsiIiq/GICo3Hj5EvjqK+Cdd4C7d4EaNYBjx4Avv8xZ00epBBYsEH9+NQRl3w8PF7cjIqLyiwGIyoXLl4FmzcTrdgkC8PHHwOnTQMOGebcNCRFPda9USbvd05OnwBMRyQUX96cyTRDES1mMHi1exNTJSZz4/Lq5PiEh4jYHD4oTnt3dxWEv9vwQEckDAxCVWQ8eiJOcd+wQ77/7LrB6tf4TmJVKoG3bkqqOiIiMGYfAqEzas0ec6LxjB2BuDnz/PbBrF8/eIiIi/bAHiMqUFy+A8ePFicoAULs2sGEDUK+epGUREVEZwwBEZcbffwO9e4vr+QDildznzAEsLaWti4iIyh4OgZHREwRg8WKgcWMx/FSsCOzcCSxaxPBDRESFwx4gMmqJicDAgcBvv4n3O3QAVq0CXF2lrYuIiMo29gCR0dq5E/DzE8OPSiX2+OzcyfBDRERFxx4gMjrPn4srOi9eLN738wM2bgTq1JG2LiIiKj/YA0RG5exZca5Pdvj5/HPgxAmGHyIiKl4MQGQUsrLEtXyaNgUuXgTc3IDdu4H58wELC6mrIyKi8oZDYCS5e/eA/v3FxQ0B4IMPgP/+F3BxkbQsIiIqx9gDRJL65Rdxjs+ePeIp7cuWAdu3M/wQEVHJYg8QSeLZM/ECpj/+KN6vX19c0blWLUnLIiIimWAPEJW606eBhg1zws+XXwLHjjH8EBFR6WEPEJWarCzgu++ACROAjAzAwwNYuxZo107qyoiISG4YgKhU3LkDhIUBf/wh3u/SBVixAnBykrYuIiKSJw6BUYnbuhWoW1cMP1ZW4hleW7cy/BARkXTYA0QlJjUVGDUKiIgQ7zduDKxfD7zxhqRlERERsQeISsbJk0CDBmL4USiA8eOBI0cYfoiIyDiwB4iKlVoNzJkDTJ4MZGYCXl7AunVAmzZSV0ZERJSDAYiKTVwc0LcvcOCAeL9HD3FhQ0dHaesiIiJ6FYfAqFhERooTnQ8cAGxsgNWrgU2bGH6IiMg4sQeIiiQlBRgxQlzPBwD8/cWJzr6+0tZFRERUEIN7gHx8fDB9+nTExcWVRD1Uhhw9Kk50XrsWMDER5/0cPMjwQ0RExs/gADRq1ChERUWhatWqeOedd7Bp0yakp6eXRG1kpLKygOnTgYAA4MYNwNtbHPqaNg0wM5O6OiIiotcrVACKjY3FiRMnUKtWLYwYMQLu7u4YPnw4Tp8+XRI1kpGZMweYMkU846tPH+DsWaBlS6mrIiIi0p9CEAShKAfIyMjADz/8gLFjxyIjIwN+fn747LPPMGDAACgUiuKqs1SlpKTA3t4eycnJsLOzk7oco3LhAtCoEfDyJbBgAfDZZ1JXREREJDLk+7vQk6AzMjKwbds2rFq1CtHR0WjWrBkGDRqEO3fu4Ouvv8bevXuxYcOGwh6ejFBmJjBggBh+3n9fnPxMRERUFhkcgE6fPo1Vq1Zh48aNMDExQb9+/fD999+jZs2amm26dOmCJk2aFGuhJL05c4C//gIcHIAffxRXeCYiIiqLDA5ATZo0wTvvvIOlS5eic+fOMNMx67VKlSro2bNnsRRIxuHCBWDqVPHnhQsBDw9JyyEiIioSgwPQjRs34O3tXeA21tbWWLVqVaGLIuOSkQH07y/+GRwMfPih1BUREREVjcFngd2/fx/Hjx/P0378+HH89ddfxVIUGZc5c4BTp8RVnTn0RURE5YHBAWjYsGG4fft2nvb4+HgMGzasWIoi43HunLi+DyAOfbm7S1sPERFRcTA4AF28eBENGzbM096gQQNcvHixWIoi45B76KtTJ3HNHyIiovLA4ACkUqmQmJiYp/3evXswNeWlxcqTb74BzpwBKlQQr+rOoS8iIiovDA5A7777LsaPH4/k5GRN25MnT/D111/jnXfeKdbiSDrnzgEzZog/L1oEuLlJWw8REVFxMrjLZt68eWjdujW8vb3RoEEDAEBsbCxcXV2xbt26Yi+QSl/uoa/OnYFevaSuiIiIqHgZHIAqVaqEc+fOYf369Th79iwsLS0xYMAA9OrVS+eaQFT2zJ6dM/S1dCmHvoiIqPwp8rXAyiM5Xwvs7FmgcWPxshcbNrD3h4iIyo5SuRbYxYsXERcXh5cvX2q1f/DBB4U9JEns5Utx6CszE+jSBeBi3kREVF4VaiXoLl264Pz581AoFMjuQMq+8rtarS7eCqnUzJoFxMYCTk4c+iIiovLN4LPARo4ciSpVquD+/fuwsrLC33//jQMHDqBx48bYv39/CZRIpSE2Fpg5U/x5yRLA1VXScoiIiEqUwT1AR48exR9//AFnZ2eYmJjAxMQErVq1wuzZs/HZZ5/hzJkzJVEnlaCXL4GwMHHoq2tXoEcPqSsiIiIqWQb3AKnVatja2gIAnJ2dcffuXQCAt7c3rly5UrzVUamYOVNc98fZGfjhBw59ERFR+WdwD9Cbb76Js2fPokqVKvD398ecOXNgbm6O5cuXo2rVqiVRI5WgM2fEuT+AOPRVsaK09RAREZUGgwPQxIkTkZaWBgCYPn063n//fQQEBMDJyQmRkZHFXiCVnNxnfXXrxqEvIiKSj2JZB+jRo0dwdHTUnAlW1sllHaDJk8XLXTg7A3//zd4fIiIq2wz5/jZoDlBGRgZMTU1x4cIFrfYKFSqUm/AjF6dP5wx9/fADww8REcmLQQHIzMwMlStX5lo/ZVx6ujj0pVYD3buLNyIiIjkx+CywCRMm4Ouvv8ajR49Koh4qBTNmAOfPAy4u4sRnIiIiuTF4EvTixYtx7do1eHh4wNvbG9bW1lqPnz59utiKo+L311/AN9+IPy9dKoYgIiIiuTE4AHXu3LkEyqDSkHvoKzRUXPSQiIhIjgwOQFOmTCmJOqgUTJ+ec7bX4sVSV0NERCSdQl8NnsqWv/4Cvv1W/HnpUvHU98JSq4GDB4F79wB3dyAgAFAqi6dOIiKi0mBwADIxMSnwlHeeIWZ8cg999ewJhIQU/lhRUcDIkcCdOzltnp7AggVFOy4REVFpMjgAbdu2Tet+RkYGzpw5gzVr1mDatGnFVhgVn2nTcoa+Fi0q/HGiosQVo19dOjM+XmzfsoUhiIiIygaDT4Pv1KmT1q1bt26YOXMm5syZgx07dhhcwJIlS+Dj4wMLCwv4+/vjxIkTBW4fHh6OGjVqwNLSEl5eXvj888/x4sWLIh2zPDt5Mmfoa9mywg99qdViz4+udcOz20aNErcjIiIydgYHoPw0a9YMMTExBu0TGRmJ0aNHY8qUKTh9+jTq1auHoKAg3L9/X+f2GzZswLhx4zBlyhRcunQJERERiIyMxNdff13oY5ZnL16IQ19ZWUDv3kCXLoU/1sGD2sNerxIE4PZtcTsiIiJjVywB6Pnz51i4cCEqVapk0H7z58/HRx99hAEDBqB27dpYtmwZrKyssHLlSp3bHzlyBC1btkTv3r3h4+ODd999F7169dLq4TH0mOXZ1KnAxYuAqyuwcGHRjnXvXvFuR0REJCWDA5CjoyMqVKiguTk6OsLW1hYrV67E3Llz9T7Oy5cvcerUKQQGBuYUY2KCwMBAHD16VOc+LVq0wKlTpzSB58aNG/jtt9/QsWPHQh+zvDp+HMj+dfz4I+DkVLTjubsX73ZERERSMngS9Pfff691FpiJiQlcXFzg7+8PR0dHvY+TlJQEtVoNV1dXrXZXV1dcvnxZ5z69e/dGUlISWrVqBUEQkJmZiU8++UQzBFaYYwJAeno60tPTNfdTUlL0fh3GKPfQV58+QKdORT9mQIB4tld8vO55QAqF+HhAQNGfi4iIqKQZHID69+9fAmXoZ//+/Zg1axZ++OEH+Pv749q1axg5ciRmzJiBSZMmFfq4s2fPLldnsE2ZAly+DLi5FX3oK5tSKZ7q3q2bGHZyh6DsPBwezvWAiIiobDB4CGzVqlXYvHlznvbNmzdjzZo1eh/H2dkZSqUSiYmJWu2JiYlwc3PTuc+kSZPQt29fDB48GH5+fujSpQtmzZqF2bNnIysrq1DHBIDx48cjOTlZc7t9+7ber8PYHDsGzJsn/vzjj0CFCsV37JAQ8VT3V6d6eXryFHgiIipbDA5As2fPhrOOc6krVqyIWbNm6X0cc3NzNGrUSOvMsaysLMTExKB58+Y693n27BlMTLRLVv5/l4MgCIU6JgCoVCrY2dlp3cqiFy+AAQPEoa8PPwQ++KD4nyMkBLh1C9i3D9iwQfzz5k2GHyIiKlsMHgKLi4tDlSpV8rR7e3sjLi7OoGONHj0aYWFhaNy4MZo2bYrw8HCkpaVhwIABAIB+/fqhUqVKmD17NgAgODgY8+fPR4MGDTRDYJMmTUJwcLAmCL3umOXZ5Mk5Q18LFpTc8yiVQNu2JXd8IiKikmZwAKpYsSLOnTsHHx8frfazZ8/CycBTjUJDQ/HgwQNMnjwZCQkJqF+/Pnbv3q2ZxBwXF6fV4zNx4kQoFApMnDgR8fHxcHFxQXBwMGbOnKn3Mcuro0eB774Tf16+vHiHvoiIiMobhSDoOqcnf2PHjkVkZCRWrVqF1q1bAwD+/PNPDBw4EN26dcO87AkoZVhKSgrs7e2RnJxcJobDnj8H6tcH/vkH6NcPMGAqFhERUblhyPe3wT1AM2bMwK1bt9CuXTuYmoq7Z2VloV+/fgbNAaLiM2mSGH48PMQzsYiIiKhgBvcAZbt69SpiY2NhaWkJPz8/eHt7F3dtkilLPUBHjgCtWomnpf/6K/Dee1JXREREJI0S7QHKVr16dVSvXr2wu1MxeP5cPOtLEICwMIYfIiIifRl8GnzXrl3xbfblxXOZM2cOunfvXixFkX4mTuTQFxERUWEYHIAOHDigufZWbh06dMCBAweKpSh6vcOHge+/F39esQJwcJC0HCIiojLF4ACUmpoKc3PzPO1mZmZl/hpaZcWzZzlDX/37AzryKBERERXA4ADk5+eHyMjIPO2bNm1C7dq1i6UoKtjEicDVq+IlKbJ7gYiIiEh/Bk+CnjRpEkJCQnD9+nW8/fbbAICYmBhs2LABW7ZsKfYCSduhQznzfTj0RUREVDgGB6Dg4GBs374ds2bNwpYtW2BpaYl69erhjz/+QAUuP1yicg99DRwIdOggdUVERERlU6HXAcqWkpKCjRs3IiIiAqdOnYJarS6u2iRjrOsAjRolXuPL0xO4cAGwt5e6IiIiIuNhyPe3wXOAsh04cABhYWHw8PDAd999h7fffhvHjh0r7OHoNQ4eBBYuFH/+738ZfoiIiIrCoCGwhIQErF69GhEREUhJSUGPHj2Qnp6O7du3cwJ0CUpLyxn6GjQICAqSuiIiIqKyTe8eoODgYNSoUQPnzp1DeHg47t69i0WLFpVkbfT/vv4auH5dHPrKvuI7ERERFZ7ePUC7du3CZ599hqFDh/ISGKXowAEOfRERERU3vXuADh06hKdPn6JRo0bw9/fH4sWLkZSUVJK1yV720BcADB7MoS8iIqLioncAatasGVasWIF79+7h448/xqZNm+Dh4YGsrCxER0fj6dOnJVmnLI0fD9y4AXh5ceiLiIioOBXpNPgrV64gIiIC69atw5MnT/DOO+9gx44dxVmfJIzhNPg//wTathV/3rMHeOcdScogIiIqM0rlNHgAqFGjBubMmYM7d+5g48aNRTkU5ZKamjP0NWQIww8REVFxK/JCiOWR1D1Aw4cDS5YAlSsD588DRrQWIxERkdEqtR4gKn779onhBwAiIhh+iIiISgIDkBFJTRWv8QUAH38MBAZKWw8REVF5xQBkRMaOBW7dAry9gblzpa6GiIio/GIAMhJ//AH88IP4c0QEYGsrbT1ERETlGQOQEUhNFa/xBQCffAK0aydtPUREROUdA5AR+OorcejLxweYM0fqaoiIiMo/BiCJxcQAS5eKP69cyaEvIiKi0sAAJKGnT3OGvj79FHjrLWnrISIikgsGIAl9+SXw779AlSrAt99KXQ0REZF8MABJZO9e4McfxZ9XrgRsbKSth4iISE4YgCSQkpIz9DVsWM5FT4mIiKh0MABJ4Msvgbg4cejrm2+kroaIiEh+GIBKWXQ0sHy5+DOHvoiIiKTBAFSKcg99jRjBoS8iIiKpMACVoq++Am7fBnx9gdmzpa6GiIhIvhiAStGnnwKNGolDX9bWUldDREQkX6ZSFyAndesCJ04AJoydREREkuJXcSlj+CEiIpIev46JiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdowiAC1ZsgQ+Pj6wsLCAv78/Tpw4ke+2bdu2hUKhyHN77733NNv0798/z+Pt27cvjZdCREREZYCp1AVERkZi9OjRWLZsGfz9/REeHo6goCBcuXIFFStWzLN9VFQUXr58qbn/8OFD1KtXD927d9farn379li1apXmvkqlKrkXQURERGWK5D1A8+fPx0cffYQBAwagdu3aWLZsGaysrLBy5Uqd21eoUAFubm6aW3R0NKysrPIEIJVKpbWdo6NjabwcIiIiKgMkDUAvX77EqVOnEBgYqGkzMTFBYGAgjh49qtcxIiIi0LNnT1hbW2u179+/HxUrVkSNGjUwdOhQPHz4MN9jpKenIyUlRetGRERE5ZekASgpKQlqtRqurq5a7a6urkhISHjt/idOnMCFCxcwePBgrfb27dtj7dq1iImJwbfffos///wTHTp0gFqt1nmc2bNnw97eXnPz8vIq/IsiIiIioyf5HKCiiIiIgJ+fH5o2barV3rNnT83Pfn5+qFu3Lnx9fbF//360a9cuz3HGjx+P0aNHa+6npKQwBBEREZVjkvYAOTs7Q6lUIjExUas9MTERbm5uBe6blpaGTZs2YdCgQa99nqpVq8LZ2RnXrl3T+bhKpYKdnZ3WjYiIiMovSQOQubk5GjVqhJiYGE1bVlYWYmJi0Lx58wL33bx5M9LT0/Hhhx++9nnu3LmDhw8fwt3dvcg1ExERUdkn+Vlgo0ePxooVK7BmzRpcunQJQ4cORVpaGgYMGAAA6NevH8aPH59nv4iICHTu3BlOTk5a7ampqfjyyy9x7Ngx3Lp1CzExMejUqROqVauGoKCgUnlNREREZNwknwMUGhqKBw8eYPLkyUhISED9+vWxe/duzcTouLg4mJho57QrV67g0KFD2LNnT57jKZVKnDt3DmvWrMGTJ0/g4eGBd999FzNmzOBaQERERAQAUAiCIEhdhLFJSUmBvb09kpOTOR+IiIiojDDk+1vyITAiIiKi0sYARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREssMARERERLLDAERERESywwBEREREsmMqdQFERCQvarUaGRkZUpdBZZCZmRmUSmWxHIsBiIiISoUgCEhISMCTJ0+kLoXKMAcHB7i5uUGhUBTpOAxARERUKrLDT8WKFWFlZVXkLzCSF0EQ8OzZM9y/fx8A4O7uXqTjMQAREVGJU6vVmvDj5OQkdTlURllaWgIA7t+/j4oVKxZpOIyToImIqMRlz/mxsrKSuBIq67I/Q0WdR8YAREREpYbDXlRUxfUZYgAiIiIqRT4+PggPD9d7+/3790OhUHDyeDHjHCAiIioz1Grg4EHg3j3A3R0ICACK6azoPF7X0zBlyhRMnTrV4OOePHkS1tbWem/fokUL3Lt3D/b29gY/F+WPAYiIiMqEqChg5Ejgzp2cNk9PYMECICSk+J/v3r17mp8jIyMxefJkXLlyRdNmY2Oj+VkQBKjVapiavv5r1cXFxaA6zM3N4ebmZtA+9HocAiMiIqMXFQV066YdfgAgPl5sj4oq/ud0c3PT3Ozt7aFQKDT3L1++DFtbW+zatQuNGjWCSqXCoUOHcP36dXTq1Amurq6wsbFBkyZNsHfvXq3jvjoEplAo8N///hddunSBlZUVqlevjh07dmgef3UIbPXq1XBwcMDvv/+OWrVqwcbGBu3bt9cKbJmZmfjss8/g4OAAJycnjB07FmFhYejcuXO+r/fhw4fo1asXKlWqBCsrK/j5+WHjxo1a22RlZWHOnDmoVq0aVCoVKleujJkzZ2oev3PnDnr16oUKFSrA2toajRs3xvHjxwvx7pc8BiAiIjJqarXY8yMIeR/Lbhs1StyutI0bNw7ffPMNLl26hLp16yI1NRUdO3ZETEwMzpw5g/bt2yM4OBhxcXEFHmfatGno0aMHzp07h44dO6JPnz549OhRvts/e/YM8+bNw7p163DgwAHExcVhzJgxmse//fZbrF+/HqtWrcLhw4eRkpKC7du3F1jDixcv0KhRI+zcuRMXLlzAkCFD0LdvX5w4cUKzzfjx4/HNN99g0qRJuHjxIjZs2ABXV1cAQGpqKtq0aYP4+Hjs2LEDZ8+exVdffYWsrCw93kkJCJRHcnKyAEBITk6WuhQionLh+fPnwsWLF4Xnz58bvO++fYIgRp2Cb/v2FXvZGqtWrRLs7e1z1bRPACBs3779tfvWqVNHWLRokea+t7e38P3332vuAxAmTpyouZ+amioAEHbt2qX1XI8fP9bUAkC4du2aZp8lS5YIrq6umvuurq7C3LlzNfczMzOFypUrC506ddL3JQuCIAjvvfee8MUXXwiCIAgpKSmCSqUSVqxYoXPbH3/8UbC1tRUePnxo0HMYqqDPkiHf35wDRERERi3XyE6xbFecGjdurHU/NTUVU6dOxc6dO3Hv3j1kZmbi+fPnr+0Bqlu3ruZna2tr2NnZaVY81sXKygq+vr6a++7u7prtk5OTkZiYiKZNm2oeVyqVaNSoUYG9MWq1GrNmzcLPP/+M+Ph4vHz5Eunp6Zp1dy5duoT09HS0a9dO5/6xsbFo0KABKlSoUOBrNRYMQEREZNT0veJBEa+MUCivns01ZswYREdHY968eahWrRosLS3RrVs3vHz5ssDjmJmZad1XKBQFhhVd2wu6xggNMHfuXCxYsADh4eHw8/ODtbU1Ro0apak9exXm/LzucWPDOUBERGTUAgLEs73yOytdoQC8vMTtpHb48GH0798fXbp0gZ+fH9zc3HDr1q1SrcHe3h6urq44efKkpk2tVuP06dMF7nf48GF06tQJH374IerVq4eqVavin3/+0TxevXp1WFpaIiYmRuf+devWRWxsbIFzl4wJAxARERk1pVI81R3IG4Ky74eHl9x6QIaoXr06oqKiEBsbi7Nnz6J3796STAIeMWIEZs+ejV9++QVXrlzByJEj8fjx4wLXNqpevTqio6Nx5MgRXLp0CR9//DESExM1j1tYWGDs2LH46quvsHbtWly/fh3Hjh1DREQEAKBXr15wc3ND586dcfjwYdy4cQNbt27F0aNHS/z1FgYDEBERGb2QEGDLFqBSJe12T0+xvSTWASqM+fPnw9HRES1atEBwcDCCgoLQsGHDUq9j7Nix6NWrF/r164fmzZvDxsYGQUFBsLCwyHefiRMnomHDhggKCkLbtm01YSa3SZMm4YsvvsDkyZNRq1YthIaGauYemZubY8+ePahYsSI6duwIPz8/fPPNN0W6YGlJUghFHTQsh1JSUmBvb4/k5GTY2dlJXQ4RUZn34sUL3Lx5E1WqVCnwS/h1SnMl6PIkKysLtWrVQo8ePTBjxgypyymSgj5Lhnx/cxI0ERGVGUol0Lat1FUYv3///Rd79uxBmzZtkJ6ejsWLF+PmzZvo3bu31KUZDQ6BERERlTMmJiZYvXo1mjRpgpYtW+L8+fPYu3cvatWqJXVpRoM9QEREROWMl5cXDh8+LHUZRo09QERERCQ7DEBEREQkOwxAREREJDsMQERERCQ7RhGAlixZAh8fH1hYWMDf3x8nTpzId9u2bdtCoVDkub333nuabQRBwOTJk+Hu7g5LS0sEBgbi6tWrpfFSiIiIqAyQPABFRkZi9OjRmDJlCk6fPo169eohKCgo36vgRkVF4d69e5rbhQsXoFQq0b17d802c+bMwcKFC7Fs2TIcP34c1tbWCAoKwosXL0rrZREREZERkzwAzZ8/Hx999BEGDBiA2rVrY9myZbCyssLKlSt1bl+hQgW4ublpbtHR0bCystIEIEEQEB4ejokTJ6JTp06oW7cu1q5di7t372L79u2l+MqIiIjEkYtRo0Zp7vv4+CA8PLzAfRQKRbF8ZxXXccojSQPQy5cvcerUKQQGBmraTExMEBgYqPfF0yIiItCzZ09YW1sDAG7evImEhAStY9rb28Pf3z/fY6anpyMlJUXrRkRE8hYcHIz27dvrfOzgwYNQKBQ4d+6cwcc9efIkhgwZUtTytEydOhX169fP037v3j106NChWJ+rvJA0ACUlJUGtVsPV1VWr3dXVFQkJCa/d/8SJE7hw4QIGDx6sacvez5Bjzp49G/b29pqbl5eXoS+FiIjKmUGDBiE6Ohp37tzJ89iqVavQuHFj1K1b1+Djuri4wMrKqjhKfC03NzeoVKpSea6yRvIhsKKIiIiAn58fmjZtWqTjjB8/HsnJyZrb7du3i6lCIiIqq95//324uLhg9erVWu2pqanYvHkzBg0ahIcPH6JXr16oVKkSrKys4Ofnh40bNxZ43FeHwK5evYrWrVvDwsICtWvXRnR0dJ59xo4dizfeeANWVlaoWrUqJk2ahIyMDADA6tWrMW3aNJw9e1ZzYlB2za8OgZ0/fx5vv/02LC0t4eTkhCFDhiA1NVXzeP/+/dG5c2fMmzcP7u7ucHJywrBhwzTPpcv169fRqVMnuLq6wsbGBk2aNMHevXu1tklPT8fYsWPh5eUFlUqFatWqISIiQvP433//jffffx92dnawtbVFQEAArl+/XuD7WFSSXgrD2dkZSqUSiYmJWu2JiYlwc3MrcN+0tDRs2rQJ06dP12rP3i8xMRHu7u5ax9TVPQgAKpWKCZmIqJQJAvDsWek/r5UVoFC8fjtTU1P069cPq1evxoQJE6D4/502b94MtVqNXr16ITU1FY0aNcLYsWNhZ2eHnTt3om/fvvD19dXrP+dZWVkICQmBq6srjh8/juTkZK35QtlsbW2xevVqeHh44Pz58/joo49ga2uLr776CqGhobhw4QJ2796tCR729vZ5jpGWloagoCA0b94cJ0+exP379zF48GAMHz5cK+Tt27cP7u7u2LdvH65du4bQ0FDUr18fH330kc7XkJqaio4dO2LmzJlQqVRYu3YtgoODceXKFVSuXBkA0K9fPxw9ehQLFy5EvXr1cPPmTSQlJQEA4uPj0bp1a7Rt2xZ//PEH7OzscPjwYWRmZr72/SsSQWJNmzYVhg8frrmvVquFSpUqCbNnzy5wv1WrVgkqlUpISkrSas/KyhLc3NyEefPmadqSk5MFlUolbNy4Ua+akpOTBQBCcnKyAa/k9TIzBWHfPkHYsEH8MzOzWA9PRGS0nj9/Lly8eFF4/vy5pi01VRDEGFS6t9RU/eu+dOmSAEDYt2+fpi0gIED48MMP893nvffeE7744gvN/TZt2ggjR47U3Pf29ha+//57QRAE4ffffxdMTU2F+Ph4zeO7du0SAAjbtm3L9znmzp0rNGrUSHN/ypQpQr169fJsl/s4y5cvFxwdHYXUXG/Azp07BRMTEyEhIUEQBEEICwsTvL29hcxcX1Ddu3cXQkND861Flzp16giLFi0SBEEQrly5IgAQoqOjdW47fvx4oUqVKsLLly/1Orauz1I2Q76/JR8CGz16NFasWIE1a9bg0qVLGDp0KNLS0jBgwAAAYmocP358nv0iIiLQuXNnODk5abUrFAqMGjUK//nPf7Bjxw6cP38e/fr1g4eHBzp37lwaL0mnqCjAxwd46y2gd2/xTx8fsZ2IiIxTzZo10aJFC82ZydeuXcPBgwcxaNAgAIBarcaMGTPg5+eHChUqwMbGBr///jvi4uL0Ov6lS5fg5eUFDw8PTVvz5s3zbBcZGYmWLVvCzc0NNjY2mDhxot7Pkfu56tWrpzlpCABatmyJrKwsXLlyRdNWp04dKJVKzX13d/d8l6YBxB6gMWPGoFatWnBwcICNjQ0uXbqkqS82NhZKpRJt2rTRuX9sbCwCAgJgZmZm0OspKsmvBh8aGooHDx5g8uTJSEhIQP369bF7927NJOa4uDiYmGjntCtXruDQoUPYs2ePzmN+9dVXSEtLw5AhQ/DkyRO0atUKu3fvhoWFRYm/Hl2iooBu3cT/e+QWHy+2b9kChIRIUhoRkWSsrIBc009K9XkNMWjQIIwYMQJLlizBqlWr4Ovrq/kynzt3LhYsWIDw8HD4+fnB2toao0aNwsuXL4ut3qNHj6JPnz6YNm0agoKCYG9vj02bNuG7774rtufI7dUgolAokJWVle/2Y8aMQXR0NObNm4dq1arB0tIS3bp107wHlpaWBT7f6x4vKZIHIAAYPnw4hg8frvOx/fv352mrUaMGhFfTRC4KhQLTp0/PMz9ICmo1MHJk3vADiG0KBTBqFNCpE5ArcBMRlXsKBZCrM8Jo9ejRAyNHjsSGDRuwdu1aDB06VDMf6PDhw+jUqRM+/PBDAOKcnn/++Qe1a9fW69i1atXC7du3ce/ePc281WPHjmltc+TIEXh7e2PChAmatn///VdrG3Nzc6jV6tc+1+rVq5GWlqbpBTp8+DBMTExQo0YNverV5fDhw+jfvz+6dOkCQOwRunXrluZxPz8/ZGVl4c8//9RaoiZb3bp1sWbNGmRkZJRqL5DkQ2Dl3cGDgI4zKDUEAbh9W9yOiIiMj42NDUJDQzF+/Hjcu3cP/fv31zxWvXp1REdH48iRI7h06RI+/vjjPCf2FCQwMBBvvPEGwsLCcPbsWRw8eFAr6GQ/R1xcHDZt2oTr169j4cKF2LZtm9Y2Pj4+uHnzJmJjY5GUlIT09PQ8z9WnTx9YWFggLCwMFy5cwL59+zBixAj07ds3z9IxhqhevTqioqIQGxuLs2fPonfv3lo9Rj4+PggLC8PAgQOxfft23Lx5E/v378fPP/8MQOwESUlJQc+ePfHXX3/h6tWrWLdundawXElgACph9+4V73ZERFT6Bg0ahMePHyMoKEhrvs7EiRPRsGFDBAUFoW3btnBzczNovqmJiQm2bduG58+fo2nTphg8eDBmzpyptc0HH3yAzz//HMOHD0f9+vVx5MgRTJo0SWubrl27on379njrrbfg4uKi81R8Kysr/P7773j06BGaNGmCbt26oV27dli8eLFhb8Yr5s+fD0dHR7Ro0QLBwcEICgpCw4YNtbZZunQpunXrhk8//RQ1a9bERx99hLS0NACAk5MT/vjjD6SmpqJNmzZo1KgRVqxYUeK9QQqhoLEkmUpJSYG9vT2Sk5NhZ2dXpGPt3y9OeH6dffuAtm2L9FREREbrxYsXuHnzJqpUqSLZfEwqHwr6LBny/c0eoBIWEAB4eua/5oRCAXh5idsRERFR6WAAKmFKJbBggfjzqyEo+354OCdAExERlSYGoFIQEiKe6l6pkna7pydPgSciIpKCUZwGLwchIeKp7gcPihOe3d3FYS/2/BAREZU+BqBSpFRyojMREZEx4BAYERGVGp54TEVVXJ8hBiAiIipx2Wu6PJPi8u9UrmR/hoq6ThCHwIiIqMQplUo4ODhoLqppZWWluZwEkT4EQcCzZ89w//59ODg4aF2wtTAYgIiIqFS4ubkBQIFXFid6HQcHB81nqSgYgIiIqFQoFAq4u7ujYsWKyMjIkLocKoPMzMyK3POTjQGIiIhKlVKpLLYvMaLC4iRoIiIikh0GICIiIpIdBiAiIiKSHc4B0iF7kaWUlBSJKyEiIiJ9ZX9v67NYIgOQDk+fPgUAeHl5SVwJERERGerp06ewt7cvcBuFwHXJ88jKysLdu3dha2vLhbrykZKSAi8vL9y+fRt2dnZSlyN7/H0YF/4+jAt/H8alJH8fgiDg6dOn8PDwgIlJwbN82AOkg4mJCTw9PaUuo0yws7PjPyhGhL8P48Lfh3Hh78O4lNTv43U9P9k4CZqIiIhkhwGIiIiIZIcBiApFpVJhypQpUKlUUpdC4O/D2PD3YVz4+zAuxvL74CRoIiIikh32ABEREZHsMAARERGR7DAAERERkewwABEREZHsMACR3mbPno0mTZrA1tYWFStWROfOnXHlyhWpy6L/980330ChUGDUqFFSlyJr8fHx+PDDD+Hk5ARLS0v4+fnhr7/+krosWVKr1Zg0aRKqVKkCS0tL+Pr6YsaMGXpdJ4qK7sCBAwgODoaHhwcUCgW2b9+u9bggCJg8eTLc3d1haWmJwMBAXL16tdTqYwAivf35558YNmwYjh07hujoaGRkZODdd99FWlqa1KXJ3smTJ/Hjjz+ibt26Upcia48fP0bLli1hZmaGXbt24eLFi/juu+/g6OgodWmy9O2332Lp0qVYvHgxLl26hG+//RZz5szBokWLpC5NFtLS0lCvXj0sWbJE5+Nz5szBwoULsWzZMhw/fhzW1tYICgrCixcvSqU+ngZPhfbgwQNUrFgRf/75J1q3bi11ObKVmpqKhg0b4ocffsB//vMf1K9fH+Hh4VKXJUvjxo3D4cOHcfDgQalLIQDvv/8+XF1dERERoWnr2rUrLC0t8dNPP0lYmfwoFAps27YNnTt3BiD2/nh4eOCLL77AmDFjAADJyclwdXXF6tWr0bNnzxKviT1AVGjJyckAgAoVKkhcibwNGzYM7733HgIDA6UuRfZ27NiBxo0bo3v37qhYsSIaNGiAFStWSF2WbLVo0QIxMTH4559/AABnz57FoUOH0KFDB4kro5s3byIhIUHr3y17e3v4+/vj6NGjpVIDL4ZKhZKVlYVRo0ahZcuWePPNN6UuR7Y2bdqE06dP4+TJk1KXQgBu3LiBpUuXYvTo0fj6669x8uRJfPbZZzA3N0dYWJjU5cnOuHHjkJKSgpo1a0KpVEKtVmPmzJno06eP1KXJXkJCAgDA1dVVq93V1VXzWEljAKJCGTZsGC5cuIBDhw5JXYps3b59GyNHjkR0dDQsLCykLocg/segcePGmDVrFgCgQYMGuHDhApYtW8YAJIGff/4Z69evx4YNG1CnTh3ExsZi1KhR8PDw4O+DOARGhhs+fDh+/fVX7Nu3D56enlKXI1unTp3C/fv30bBhQ5iamsLU1BR//vknFi5cCFNTU6jVaqlLlB13d3fUrl1bq61WrVqIi4uTqCJ5+/LLLzFu3Dj07NkTfn5+6Nu3Lz7//HPMnj1b6tJkz83NDQCQmJio1Z6YmKh5rKQxAJHeBEHA8OHDsW3bNvzxxx+oUqWK1CXJWrt27XD+/HnExsZqbo0bN0afPn0QGxsLpVIpdYmy07JlyzxLQ/zzzz/w9vaWqCJ5e/bsGUxMtL/mlEolsrKyJKqIslWpUgVubm6IiYnRtKWkpOD48eNo3rx5qdTAITDS27Bhw7Bhwwb88ssvsLW11YzT2tvbw9LSUuLq5MfW1jbP/Ctra2s4OTlxXpZEPv/8c7Ro0QKzZs1Cjx49cOLECSxfvhzLly+XujRZCg4OxsyZM1G5cmXUqVMHZ86cwfz58zFw4ECpS5OF1NRUXLt2TXP/5s2biI2NRYUKFVC5cmWMGjUK//nPf1C9enVUqVIFkyZNgoeHh+ZMsRInEOkJgM7bqlWrpC6N/l+bNm2EkSNHSl2GrP3vf/8T3nzzTUGlUgk1a9YUli9fLnVJspWSkiKMHDlSqFy5smBhYSFUrVpVmDBhgpCeni51abKwb98+nd8ZYWFhgiAIQlZWljBp0iTB1dVVUKlUQrt27YQrV66UWn1cB4iIiIhkh3OAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiIiEh2GICIiIhIdhiAiIiISHYYgIiI8qFQKLB9+3apyyCiEsAARERGqX///lAoFHlu7du3l7o0IioHeC0wIjJa7du3x6pVq7TaVCqVRNUQUXnCHiAiMloqlQpubm5aN0dHRwDi8NTSpUvRoUMHWFpaomrVqtiyZYvW/ufPn8fbb78NS0tLODk5YciQIUhNTdXaZuXKlahTpw5UKhXc3d0xfPhwrceTkpLQpUsXWFlZoXr16tixY4fmscePH6NPnz5wcXGBpaUlqlevniewEZFxYgAiojJr0qRJ6Nq1K86ePYs+ffqgZ8+euHTpEgAgLS0NQUFBcHR0xMmTJ7F582bs3btXK+AsXboUw4YNw5AhQ3D+/Hns2LED1apV03qOadOmoUePHjh37hw6duyIPn364NGjR5rnv3jxInbt2oVLly5h6dKlcHZ2Lr03gIgKr9Quu0pEZICwsDBBqVQK1tbWWreZM2cKgiAIAIRPPvlEax9/f39h6NChgiAIwvLlywVHR0chNTVV8/jOnTsFExMTISEhQRAEQfDw8BAmTJiQbw0AhIkTJ2rup6amCgCEXbt2CYIgCMHBwcKAAQOK5wUTUaniHCAiMlpvvfUWli5dqtVWoUIFzc/NmzfXeqx58+aIjY0FAFy6dAn16tWDtbW15vGWLVsiKysLV65cgUKhwN27d9GuXbsCa6hbt67mZ2tra9jZ2eH+/fsAgKFDh6Jr1644ffo03n33XXTu3BktWrQo1GslotLFAERERsva2jrPkFRxsbS01Gs7MzMzrfsKhQJZWVkAgA4dOuDff//Fb7/9hujoaLRr1w7Dhg3DvHnzir1eIipenANERGXWsWPH8tyvVasWAKBWrVo4e/Ys0tLSNI8fPnwYJiYmqFGjBmxtbeHj44OYmJgi1eDi4oKwsDD89NNPCA8Px/Lly4t0PCIqHewBIiKjlZ6ejoSEBK02U1NTzUTjzZs3o3HjxmjVqhXWr1+PEydOICIiAgDQp08fTJkyBWFhYZg6dSoePHiAESNGoG/fvnB1dQUATJ06FZ988gkqVqyIDh064OnTpzh8+DBGjBihV32TJ09Go0aNUKdOHaSnp+PXX3/VBDAiMm4MQERktHbv3g13d3ettho1auDy5csAxDO0Nm3ahE8//RTu7u7YuHEjateuDQCwsrLC77//jpEjR6JJkyawsrJC165dMX/+fM2xwsLC8OLFC3z//fcYM2YMnJ2d0a1bN73rMzc3x/jx43Hr1i1YWloiICAAmzZtKoZXTkQlTSEIgiB1EUREhlIoFNi2bRs6d+4sdSlEVAZxDhARERHJDgMQERERyQ7nABFRmcTReyIqCvYAERERkewwABEREZHsMAARERGR7DAAERERkewwABEREZHsMAARERGR7DAAERERkewwABEREZHsMAARERGR7Pwfdn0MxQovG2kAAAAASUVORK5CYII=", - "text/plain": [ - "
" - ] - }, - "metadata": {}, - "output_type": "display_data" - } - ], - "source": [ - "plt.plot(epochs, acc, 'bo', label='Training acc')\n", - "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n", - "plt.title('Training and validation accuracy')\n", - "plt.xlabel('Epochs')\n", - "plt.ylabel('Accuracy')\n", - "plt.legend(loc='lower right')\n", - "\n", - "plt.show()" - ] - }, - { - "cell_type": "code", - "execution_count": 29, - "id": "93b0a42c-437e-41bb-99e7-d58cb8036a3a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "782/782 [==============================] - 5s 6ms/step - loss: 0.3103 - accuracy: 0.8737\n", - "0.8736799955368042\n" - ] - } - ], - "source": [ - "export_model = tf.keras.Sequential([\n", - " vectorize_layer,\n", - " model,\n", - " layers.Activation('sigmoid')\n", - "])\n", - "\n", - "export_model.compile(\n", - " loss=losses.BinaryCrossentropy(from_logits=False), optimizer=\"adam\", metrics=['accuracy']\n", - ")\n", - "\n", - "# Test it with `raw_test_ds`, which yields raw strings\n", - "loss, accuracy = export_model.evaluate(raw_test_ds)\n", - "print(accuracy)" - ] - }, - { - "cell_type": "code", - "execution_count": 30, - "id": "8939539b-a600-48b1-a55e-3f1087f4a855", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "1/1 [==============================] - 0s 99ms/step\n" - ] - }, - { - "data": { - "text/plain": [ - "array([[0.6167954],\n", - " [0.4397881],\n", - " [0.3572815]], dtype=float32)" - ] - }, - "execution_count": 30, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "examples = [\n", - " \"The movie was great!\",\n", - " \"The movie was okay.\",\n", - " \"The movie was terrible...\"\n", - "]\n", - "\n", - "export_model.predict(examples)" - ] - }, - { - "cell_type": "markdown", - "id": "f6b40a59-8d3b-44ec-a4f7-92c5742a0c1c", - "metadata": {}, - "source": [ - "### Save Model" - ] - }, - { - "cell_type": "code", - "execution_count": 31, - "id": "e2aa1770-2bf9-436b-af32-16b64fc97490", - "metadata": {}, - "outputs": [], - "source": [ - "!rm -rf text_model" - ] - }, - { - "cell_type": "code", - "execution_count": 32, - "id": "7f22cc32-2708-4808-8e76-99024da87a21", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "WARNING:absl:Found untraced functions such as _update_step_xla, _update_step_xla while saving (showing 2 of 2). These functions will not be directly callable after loading.\n" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "INFO:tensorflow:Assets written to: text_model/assets\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - "INFO:tensorflow:Assets written to: text_model/assets\n" - ] - } - ], - "source": [ - "export_model.save('text_model')" - ] - }, - { - "cell_type": "markdown", - "id": "3add8a92-bb90-415b-ae21-417a6b722a8b", - "metadata": {}, - "source": [ - "### Inspect saved model" - ] - }, - { - "cell_type": "code", - "execution_count": 33, - "id": "d5dda184-8288-4de2-95a9-41d0a9744b2c", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "\u001b[01;34mtext_model\u001b[00m\n", - "├── \u001b[01;34massets\u001b[00m\n", - "├── fingerprint.pb\n", - "├── keras_metadata.pb\n", - "├── saved_model.pb\n", - "└── \u001b[01;34mvariables\u001b[00m\n", - " ├── variables.data-00000-of-00001\n", - " └── variables.index\n", - "\n", - "2 directories, 5 files\n" - ] - } - ], - "source": [ - "!tree text_model" - ] - }, - { - "cell_type": "code", - "execution_count": 34, - "id": "e756f01d-dcef-4132-a0e6-4231792356de", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "The given SavedModel SignatureDef contains the following input(s):\n", - " inputs['text_vectorization_input'] tensor_info:\n", - " dtype: DT_STRING\n", - " shape: (-1)\n", - " name: serving_default_text_vectorization_input:0\n", - "The given SavedModel SignatureDef contains the following output(s):\n", - " outputs['activation'] tensor_info:\n", - " dtype: DT_FLOAT\n", - " shape: (-1, 1)\n", - " name: StatefulPartitionedCall:0\n", - "Method name is: tensorflow/serving/predict\n" - ] - } - ], - "source": [ - "!saved_model_cli show --dir text_model --tag_set serve --signature_def serving_default" - ] - }, - { - "cell_type": "markdown", - "id": "e0461f74-fdd0-4f30-9f44-0be7ad00d9b0", - "metadata": {}, - "source": [ - "### Load model" - ] - }, - { - "cell_type": "code", - "execution_count": 35, - "id": "c9cf2c7f-5e86-4ff8-984e-dd0ed7a3ece9", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "Model: \"sequential_1\"\n", - "_________________________________________________________________\n", - " Layer (type) Output Shape Param # \n", - "=================================================================\n", - " text_vectorization (TextVec (None, 250) 0 \n", - " torization) \n", - " \n", - " sequential (Sequential) (None, 1) 160033 \n", - " \n", - " activation (Activation) (None, 1) 0 \n", - " \n", - "=================================================================\n", - "Total params: 160,033\n", - "Trainable params: 160,033\n", - "Non-trainable params: 0\n", - "_________________________________________________________________\n" - ] - } - ], - "source": [ - "# register callables as custom objects before loading\n", - "custom_objects = {\"vectorize_layer\": vectorize_layer, \"custom_standardization\": custom_standardization}\n", - "with tf.keras.utils.custom_object_scope(custom_objects):\n", - " new_model = tf.keras.models.load_model('text_model')\n", - "\n", - "new_model.summary()" - ] - }, - { - "cell_type": "markdown", - "id": "242a4f7e-fa45-4d21-b103-fe3718bc0f10", - "metadata": {}, - "source": [ - "### Predict" - ] - }, - { - "cell_type": "code", - "execution_count": 36, - "id": "531680b2-42ef-4205-9a38-6995aee9f340", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "1/1 [==============================] - 0s 69ms/step\n" - ] - }, - { - "data": { - "text/plain": [ - "array([[0.6167954],\n", - " [0.4397881],\n", - " [0.3572815]], dtype=float32)" - ] - }, - "execution_count": 36, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "new_model.predict(examples)" - ] - }, - { - "cell_type": "markdown", - "id": "a82ae387-1587-4175-b4b2-66586e4668f7", - "metadata": {}, - "source": [ - "## PySpark" - ] - }, - { - "cell_type": "markdown", - "id": "0b5ac416-a37f-4f0b-8c77-628f0fcede69", - "metadata": {}, - "source": [ - "## Inference using Spark DL API\n", - "Note: you can restart the kernel and run from this point to simulate running in a different node or environment." - ] - }, - { - "cell_type": "code", - "execution_count": 37, - "id": "d6d515c2-ce53-4af5-a936-ae91fdecea99", - "metadata": {}, - "outputs": [], - "source": [ - "import os\n", - "from pyspark.ml.functions import predict_batch_udf\n", - "from pyspark.sql.functions import struct, col\n", - "from pyspark.sql.types import ArrayType, FloatType, DoubleType" - ] - }, - { - "cell_type": "code", - "execution_count": 38, - "id": "6d0d567a-0ef8-4a93-8235-e89ace2c82ad", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - "[Stage 1:> (0 + 1) / 1]\r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+----------------------------------------------------------------------------------------------------+\n", - "| lines|\n", - "+----------------------------------------------------------------------------------------------------+\n", - "|...But not this one! I always wanted to know \"what happened\" next. We will never know for sure wh...|\n", - "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story without the...|\n", - "|I watched this movie to see the direction one of the most promising young talents in movies was g...|\n", - "|This movie makes you wish imdb would let you vote a zero. One of the two movies I've ever walked ...|\n", - "|I never want to see this movie again!

Not only is it dreadfully bad, but I can't stand...|\n", - "|(As a note, I'd like to say that I saw this movie at my annual church camp, where the entire yout...|\n", - "|Don't get me wrong, I love the TV series of League Of Gentlemen. It was funny, twisted and comple...|\n", - "|Did you ever think, like after watching a horror movie with a group of friends: \"Wow, this is so ...|\n", - "|Awful, awful, awful...

I loved the original film. It was funny, charming, and had hear...|\n", - "|This movie seems a little clunky around the edges, like not quite enough zaniness was thrown it w...|\n", - "|I rented this movie hoping that it would provide some good entertainment and some cool poker know...|\n", - "|Well, where to start describing this celluloid debacle? You already know the big fat NADA passing...|\n", - "|I hoped for this show to be somewhat realistic. It stroke me as just another mainstream show afte...|\n", - "|All I have to say is one word...SUCKS!!!!. The only reason I gave this a 2 is because Josh Hartne...|\n", - "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay.

...|\n", - "|This critique tells the story of 4 little friends who went to watch Angels and Demons the movie o...|\n", - "|This review contains a partial spoiler.

Shallow from the outset, 'D.O.A.' at least sta...|\n", - "|I'm rather surprised that anybody found this film touching or moving.

The basic premis...|\n", - "|If you like bad movies (and you must to watch this one) here's a good one. Not quite as funny as ...|\n", - "|This is really bad, the characters were bland, the story was boring, and there is no sex scene. F...|\n", - "+----------------------------------------------------------------------------------------------------+\n", - "only showing top 20 rows\n", - "\n" - ] - }, - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - } - ], - "source": [ - "# note: using IMDB parquet dataset from huggingface/conditional_generation.ipynb\n", - "df = spark.read.parquet(\"../huggingface/imdb_test\")\n", - "df.show(truncate=100)" - ] - }, - { - "cell_type": "code", - "execution_count": 39, - "id": "7b7a8395-e2ae-4c3c-bf57-763dfde600ad", - "metadata": {}, - "outputs": [], - "source": [ - "text_model_path = \"{}/text_model\".format(os.getcwd())" - ] - }, - { - "cell_type": "code", - "execution_count": 40, - "id": "8c0524cf-3a75-4fb8-8025-f0654acce13e", - "metadata": {}, - "outputs": [], - "source": [ - "def predict_batch_fn():\n", - " # since this function runs on the executor, any required imports should be added inside the function.\n", - " import re\n", - " import string\n", - " import tensorflow as tf\n", - " from tensorflow.keras import layers\n", - "\n", - " def custom_standardization(input_data):\n", - " lowercase = tf.strings.lower(input_data)\n", - " stripped_html = tf.strings.regex_replace(lowercase, \"
\", \" \")\n", - " return tf.strings.regex_replace(\n", - " stripped_html, \"[%s]\" % re.escape(string.punctuation), \"\"\n", - " )\n", - "\n", - " max_features = 10000\n", - " sequence_length = 250\n", - "\n", - " vectorize_layer = layers.TextVectorization(\n", - " standardize=custom_standardization,\n", - " max_tokens=max_features,\n", - " output_mode=\"int\",\n", - " output_sequence_length=sequence_length,\n", - " )\n", - "\n", - " custom_objects = {\"vectorize_layer\": vectorize_layer,\n", - " \"custom_standardization\": custom_standardization}\n", - " with tf.keras.utils.custom_object_scope(custom_objects):\n", - " model = tf.keras.models.load_model(text_model_path)\n", - "\n", - " def predict(inputs):\n", - " return model.predict(inputs)\n", - "\n", - " return predict" - ] - }, - { - "cell_type": "code", - "execution_count": 41, - "id": "0d603644-d938-4c87-aa8a-2512251638d5", - "metadata": {}, - "outputs": [], - "source": [ - "classify = predict_batch_udf(predict_batch_fn,\n", - " return_type=FloatType(),\n", - " batch_size=256)" - ] - }, - { - "cell_type": "code", - "execution_count": 42, - "id": "0b480622-8dc1-4879-933e-c43112768630", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 269 ms, sys: 76.7 ms, total: 345 ms\n", - "Wall time: 6.81 s\n" - ] - } - ], - "source": [ - "%%time\n", - "predictions = df.withColumn(\"preds\", classify(struct(\"lines\")))\n", - "results = predictions.collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 43, - "id": "31b0a262-387e-4a5e-a60e-b9b8ee456199", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 106 ms, sys: 10 ms, total: 116 ms\n", - "Wall time: 2.47 s\n" - ] - } - ], - "source": [ - "%%time\n", - "predictions = df.withColumn(\"preds\", classify(\"lines\"))\n", - "results = predictions.collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 44, - "id": "7ef9e431-59f5-4b29-9f79-ae16a9cfb0b9", - "metadata": {}, - "outputs": [ - { - "name": "stderr", - "output_type": "stream", - "text": [ - " \r" - ] - }, - { - "name": "stdout", - "output_type": "stream", - "text": [ - "CPU times: user 93.7 ms, sys: 21.3 ms, total: 115 ms\n", - "Wall time: 2.51 s\n" - ] - } - ], - "source": [ - "%%time\n", - "predictions = df.withColumn(\"preds\", classify(col(\"lines\")))\n", - "results = predictions.collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 45, - "id": "9a325ee2-3268-414a-bb75-a5fcf794f512", - "metadata": { - "scrolled": true - }, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------------------------------------------------------------------------------+------------+\n", - "| lines| preds|\n", - "+--------------------------------------------------------------------------------+------------+\n", - "|...But not this one! I always wanted to know \"what happened\" next. We will ne...| 0.55854315|\n", - "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pi...| 0.20151669|\n", - "|I watched this movie to see the direction one of the most promising young tal...| 0.2886543|\n", - "|This movie makes you wish imdb would let you vote a zero. One of the two movi...| 0.4254548|\n", - "|I never want to see this movie again!

Not only is it dreadfully ba...| 0.007963314|\n", - "|(As a note, I'd like to say that I saw this movie at my annual church camp, w...| 0.52016854|\n", - "|Don't get me wrong, I love the TV series of League Of Gentlemen. It was funny...|3.7298937E-4|\n", - "|Did you ever think, like after watching a horror movie with a group of friend...|0.0073868474|\n", - "|Awful, awful, awful...

I loved the original film. It was funny, ch...|0.0020545013|\n", - "|This movie seems a little clunky around the edges, like not quite enough zani...| 0.19478825|\n", - "|I rented this movie hoping that it would provide some good entertainment and ...| 0.010234797|\n", - "|Well, where to start describing this celluloid debacle? You already know the ...| 0.02019683|\n", - "|I hoped for this show to be somewhat realistic. It stroke me as just another ...| 0.39049864|\n", - "|All I have to say is one word...SUCKS!!!!. The only reason I gave this a 2 is...| 0.19343555|\n", - "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy scr...| 3.712686E-5|\n", - "|This critique tells the story of 4 little friends who went to watch Angels an...| 0.16827627|\n", - "|This review contains a partial spoiler.

Shallow from the outset, '...| 0.043480344|\n", - "|I'm rather surprised that anybody found this film touching or moving.
>>> containers: {}\".format([c.short_id for c in containers]))\n", - " else:\n", - " container=client.containers.run(\n", - " \"nvcr.io/nvidia/tritonserver:23.04-py3\", \"tritonserver --model-repository=/models\",\n", - " detach=True,\n", - " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", - " name=\"spark-triton\",\n", - " network_mode=\"host\",\n", - " remove=True,\n", - " shm_size=\"128M\",\n", - " volumes={\n", - " triton_models_dir: {\"bind\": \"/models\", \"mode\": \"ro\"},\n", - " text_model_dir: {\"bind\": \"/text_model\", \"mode\": \"ro\"}\n", - " }\n", - " )\n", - " print(\">>>> starting triton: {}\".format(container.short_id))\n", - "\n", - " # wait for triton to be running\n", - " time.sleep(15)\n", - " client = grpcclient.InferenceServerClient(\"localhost:8001\")\n", - " ready = False\n", - " while not ready:\n", - " try:\n", - " ready = client.is_server_ready()\n", - " except Exception as e:\n", - " time.sleep(5)\n", - " \n", - " return [True]\n", - "\n", - "nodeRDD.barrier().mapPartitions(start_triton).collect()" - ] - }, - { - "cell_type": "markdown", - "id": "287873da-6202-4b55-97fb-cda8644b1fee", - "metadata": {}, - "source": [ - "#### Run inference" - ] - }, - { - "cell_type": "code", - "execution_count": 49, - "id": "41106a02-236e-4cb3-ac51-76aa64b663c2", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+----------------------------------------------------------------------------------------------------+\n", - "| lines|\n", - "+----------------------------------------------------------------------------------------------------+\n", - "|...But not this one! I always wanted to know \"what happened\" next. We will never know for sure wh...|\n", - "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pimp story without the...|\n", - "|I watched this movie to see the direction one of the most promising young talents in movies was g...|\n", - "|This movie makes you wish imdb would let you vote a zero. One of the two movies I've ever walked ...|\n", - "|I never want to see this movie again!

Not only is it dreadfully bad, but I can't stand...|\n", - "|(As a note, I'd like to say that I saw this movie at my annual church camp, where the entire yout...|\n", - "|Don't get me wrong, I love the TV series of League Of Gentlemen. It was funny, twisted and comple...|\n", - "|Did you ever think, like after watching a horror movie with a group of friends: \"Wow, this is so ...|\n", - "|Awful, awful, awful...

I loved the original film. It was funny, charming, and had hear...|\n", - "|This movie seems a little clunky around the edges, like not quite enough zaniness was thrown it w...|\n", - "|I rented this movie hoping that it would provide some good entertainment and some cool poker know...|\n", - "|Well, where to start describing this celluloid debacle? You already know the big fat NADA passing...|\n", - "|I hoped for this show to be somewhat realistic. It stroke me as just another mainstream show afte...|\n", - "|All I have to say is one word...SUCKS!!!!. The only reason I gave this a 2 is because Josh Hartne...|\n", - "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy screenplay.

...|\n", - "|This critique tells the story of 4 little friends who went to watch Angels and Demons the movie o...|\n", - "|This review contains a partial spoiler.

Shallow from the outset, 'D.O.A.' at least sta...|\n", - "|I'm rather surprised that anybody found this film touching or moving.

The basic premis...|\n", - "|If you like bad movies (and you must to watch this one) here's a good one. Not quite as funny as ...|\n", - "|This is really bad, the characters were bland, the story was boring, and there is no sex scene. F...|\n", - "+----------------------------------------------------------------------------------------------------+\n", - "only showing top 20 rows\n", - "\n" - ] - } - ], - "source": [ - "# note: using IMDB parquet dataset from huggingface/conditional_generation.ipynb\n", - "df = spark.read.parquet(\"../huggingface/imdb_test\").repartition(1)\n", - "df.show(truncate=100)" - ] - }, - { - "cell_type": "code", - "execution_count": 50, - "id": "8b763167-7f50-4278-9bc9-6c3433b62294", - "metadata": {}, - "outputs": [ - { - "data": { - "text/plain": [ - "['lines']" - ] - }, - "execution_count": 50, - "metadata": {}, - "output_type": "execute_result" - } - ], - "source": [ - "columns = df.columns\n", - "columns" - ] - }, - { - "cell_type": "code", - "execution_count": 51, - "id": "29b0cc0d-c480-4e4a-bd41-207dc314cba5", - "metadata": {}, - "outputs": [], - "source": [ - "def triton_fn(triton_uri, model_name):\n", - " import numpy as np\n", - " import tritonclient.grpc as grpcclient\n", - " \n", - " np_types = {\n", - " \"BOOL\": np.dtype(np.bool8),\n", - " \"INT8\": np.dtype(np.int8),\n", - " \"INT16\": np.dtype(np.int16),\n", - " \"INT32\": np.dtype(np.int32),\n", - " \"INT64\": np.dtype(np.int64),\n", - " \"FP16\": np.dtype(np.float16),\n", - " \"FP32\": np.dtype(np.float32),\n", - " \"FP64\": np.dtype(np.float64),\n", - " \"FP64\": np.dtype(np.double),\n", - " \"BYTES\": np.dtype(object)\n", - " }\n", - "\n", - " client = grpcclient.InferenceServerClient(triton_uri)\n", - " model_meta = client.get_model_metadata(model_name)\n", - " \n", - " def predict(inputs):\n", - " if isinstance(inputs, np.ndarray):\n", - " # single ndarray input\n", - " request = [grpcclient.InferInput(model_meta.inputs[0].name, inputs.shape, model_meta.inputs[0].datatype)]\n", - " request[0].set_data_from_numpy(inputs.astype(np_types[model_meta.inputs[0].datatype]))\n", - " else:\n", - " # dict of multiple ndarray inputs\n", - " request = [grpcclient.InferInput(i.name, inputs[i.name].shape, i.datatype) for i in model_meta.inputs]\n", - " for i in request:\n", - " i.set_data_from_numpy(inputs[i.name()].astype(np_types[i.datatype()]))\n", - " \n", - " response = client.infer(model_name, inputs=request)\n", - " \n", - " if len(model_meta.outputs) > 1:\n", - " # return dictionary of numpy arrays\n", - " return {o.name: response.as_numpy(o.name) for o in model_meta.outputs}\n", - " else:\n", - " # return single numpy array\n", - " return response.as_numpy(model_meta.outputs[0].name)\n", - " \n", - " return predict" - ] - }, - { - "cell_type": "code", - "execution_count": 52, - "id": "8e06d33f-5cef-4a48-afc3-5d468f8ec2b4", - "metadata": {}, - "outputs": [], - "source": [ - "from functools import partial\n", - "\n", - "classify = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"text_classification\"),\n", - " input_tensor_shapes=[[1]],\n", - " return_type=FloatType(),\n", - " batch_size=2048)" - ] - }, - { - "cell_type": "code", - "execution_count": 53, - "id": "d89e74ad-e551-4bfa-ad08-98725878630a", - "metadata": {}, - "outputs": [ - { - "name": "stdout", - "output_type": "stream", - "text": [ - "+--------------------------------------------------------------------------------+------------+\n", - "| lines| preds|\n", - "+--------------------------------------------------------------------------------+------------+\n", - "|...But not this one! I always wanted to know \"what happened\" next. We will ne...| 0.55854315|\n", - "|Hard up, No proper jobs going down at the pit, why not rent your kids! DIY pi...| 0.20151669|\n", - "|I watched this movie to see the direction one of the most promising young tal...| 0.2886543|\n", - "|This movie makes you wish imdb would let you vote a zero. One of the two movi...| 0.4254548|\n", - "|I never want to see this movie again!

Not only is it dreadfully ba...| 0.007963314|\n", - "|(As a note, I'd like to say that I saw this movie at my annual church camp, w...| 0.52016854|\n", - "|Don't get me wrong, I love the TV series of League Of Gentlemen. It was funny...|3.7298937E-4|\n", - "|Did you ever think, like after watching a horror movie with a group of friend...|0.0073868474|\n", - "|Awful, awful, awful...

I loved the original film. It was funny, ch...|0.0020545013|\n", - "|This movie seems a little clunky around the edges, like not quite enough zani...| 0.19478825|\n", - "|I rented this movie hoping that it would provide some good entertainment and ...| 0.010234797|\n", - "|Well, where to start describing this celluloid debacle? You already know the ...| 0.02019683|\n", - "|I hoped for this show to be somewhat realistic. It stroke me as just another ...| 0.39049864|\n", - "|All I have to say is one word...SUCKS!!!!. The only reason I gave this a 2 is...| 0.19343555|\n", - "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy scr...| 3.712686E-5|\n", - "|This critique tells the story of 4 little friends who went to watch Angels an...| 0.16827627|\n", - "|This review contains a partial spoiler.

Shallow from the outset, '...| 0.043480344|\n", - "|I'm rather surprised that anybody found this film touching or moving.

Not only is it dreadfully ba...| 0.007963314|\n", - "|(As a note, I'd like to say that I saw this movie at my annual church camp, w...| 0.52016854|\n", - "|Don't get me wrong, I love the TV series of League Of Gentlemen. It was funny...|3.7298937E-4|\n", - "|Did you ever think, like after watching a horror movie with a group of friend...|0.0073868474|\n", - "|Awful, awful, awful...

I loved the original film. It was funny, ch...|0.0020545013|\n", - "|This movie seems a little clunky around the edges, like not quite enough zani...| 0.19478825|\n", - "|I rented this movie hoping that it would provide some good entertainment and ...| 0.010234797|\n", - "|Well, where to start describing this celluloid debacle? You already know the ...| 0.02019683|\n", - "|I hoped for this show to be somewhat realistic. It stroke me as just another ...| 0.39049864|\n", - "|All I have to say is one word...SUCKS!!!!. The only reason I gave this a 2 is...| 0.19343555|\n", - "|Honestly awful film, bad editing, awful lighting, dire dialog and scrappy scr...| 3.712686E-5|\n", - "|This critique tells the story of 4 little friends who went to watch Angels an...| 0.16827627|\n", - "|This review contains a partial spoiler.

Shallow from the outset, '...| 0.043480344|\n", - "|I'm rather surprised that anybody found this film touching or moving.
>>> stopping containers: {}\".format([c.short_id for c in containers]))\n", - " if containers:\n", - " container=containers[0]\n", - " container.stop(timeout=120)\n", - "\n", - " return [True]\n", - "\n", - "nodeRDD.barrier().mapPartitions(stop_triton).collect()" - ] - }, - { - "cell_type": "code", - "execution_count": 56, - "id": "54a90574-7cbb-487b-b7a8-dcda0e6e301f", - "metadata": {}, - "outputs": [], - "source": [ - "spark.stop()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "88e3bfea-a825-46eb-b8c2-921a932c0089", - "metadata": {}, - "outputs": [], - "source": [] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.9.10" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/text_classification_tf.ipynb b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/text_classification_tf.ipynb new file mode 100644 index 000000000..971bf393d --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/tensorflow/text_classification_tf.ipynb @@ -0,0 +1,1850 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "2cd2accf-5877-4136-a243-7a33a13ce2b4", + "metadata": {}, + "source": [ + "# Pyspark TensorFlow Inference\n", + "\n", + "## Text classification\n", + "Based on: https://www.tensorflow.org/tutorials/keras/text_classification" + ] + }, + { + "cell_type": "markdown", + "id": "bc72d0ed", + "metadata": {}, + "source": [ + "### Using TensorFlow\n", + "Note that cuFFT/cuDNN/cuBLAS registration errors are expected with `tf=2.17.0` and will not affect behavior, as noted in [this issue.](https://github.com/tensorflow/tensorflow/issues/62075) \n", + "This notebook does not demonstrate inference with TensorRT, as [TF-TRT](https://docs.nvidia.com/deeplearning/tensorrt/release-notes/index.html#tensorrt-10) does not yet support `tf=2.17.0`. See the `pytorch` notebooks for TensorRT demos." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "76f0f5df-502f-444e-b2ee-1122e1dea870", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:43:56.140645: I tensorflow/core/util/port.cc:153] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.\n", + "2024-10-03 17:43:56.147227: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered\n", + "2024-10-03 17:43:56.154601: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered\n", + "2024-10-03 17:43:56.156763: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered\n", + "2024-10-03 17:43:56.162424: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n", + "To enable the following instructions: AVX2 AVX_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n", + "2024-10-03 17:43:56.485452: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n" + ] + } + ], + "source": [ + "import os\n", + "import re\n", + "import shutil\n", + "import string\n", + "\n", + "import matplotlib.pyplot as plt\n", + "import tensorflow as tf\n", + "from tensorflow.keras import layers, losses" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "a364ad5f-b269-45b5-ab8b-d8f34fb642b7", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "2.17.0\n" + ] + } + ], + "source": [ + "print(tf.__version__)" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "57b1d71f", + "metadata": {}, + "outputs": [], + "source": [ + "# Enable GPU memory growth\n", + "gpus = tf.config.experimental.list_physical_devices('GPU')\n", + "if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "d229c1b6-3967-46b5-9ea8-68f4b42dd211", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Downloading data from https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\n", + "\u001b[1m84125825/84125825\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 0us/step\n" + ] + } + ], + "source": [ + "url = \"https://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz\"\n", + "\n", + "dataset = tf.keras.utils.get_file(\n", + " \"aclImdb_v1\", url, untar=True, cache_dir=\".\", cache_subdir=\"\"\n", + ")\n", + "\n", + "dataset_dir = os.path.join(os.path.dirname(dataset), \"aclImdb\")" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "1f8038ae-8bc1-46bf-ae4c-6da08886c473", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['README', 'imdb.vocab', 'test', 'train', 'imdbEr.txt']" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "os.listdir(dataset_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "12faaa3f-3441-4361-b9eb-4317e8c2c2f7", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['pos',\n", + " 'labeledBow.feat',\n", + " 'urls_pos.txt',\n", + " 'neg',\n", + " 'urls_unsup.txt',\n", + " 'unsupBow.feat',\n", + " 'urls_neg.txt',\n", + " 'unsup']" + ] + }, + "execution_count": 7, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "train_dir = os.path.join(dataset_dir, \"train\")\n", + "os.listdir(train_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "152cc0cc-65d0-4e17-9ee8-222390df45b5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Rachel Griffiths writes and directs this award winning short film. A heartwarming story about coping with grief and cherishing the memory of those we've loved and lost. Although, only 15 minutes long, Griffiths manages to capture so much emotion and truth onto film in the short space of time. Bud Tingwell gives a touching performance as Will, a widower struggling to cope with his wife's death. Will is confronted by the harsh reality of loneliness and helplessness as he proceeds to take care of Ruth's pet cow, Tulip. The film displays the grief and responsibility one feels for those they have loved and lost. Good cinematography, great direction, and superbly acted. It will bring tears to all those who have lost a loved one, and survived.\n" + ] + } + ], + "source": [ + "sample_file = os.path.join(train_dir, \"pos/1181_9.txt\")\n", + "with open(sample_file) as f:\n", + " print(f.read())" + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "b2277f58-78c8-4a12-bc98-5103e7c81a35", + "metadata": {}, + "outputs": [], + "source": [ + "remove_dir = os.path.join(train_dir, \"unsup\")\n", + "shutil.rmtree(remove_dir)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "ed83de92-ebb3-4170-b2bf-25265c6a6942", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Found 25000 files belonging to 2 classes.\n", + "Using 20000 files for training.\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:44:07.678162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:2021] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 44790 MB memory: -> device: 0, name: NVIDIA RTX A6000, pci bus id: 0000:01:00.0, compute capability: 8.6\n" + ] + } + ], + "source": [ + "batch_size = 32\n", + "seed = 42\n", + "\n", + "raw_train_ds = tf.keras.utils.text_dataset_from_directory(\n", + " \"aclImdb/train\",\n", + " batch_size=batch_size,\n", + " validation_split=0.2,\n", + " subset=\"training\",\n", + " seed=seed,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "57c30568-daa8-4b2b-b30a-577c984a8af5", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Review b'\"Pandemonium\" is a horror movie spoof that comes off more stupid than funny. Believe me when I tell you, I love comedies. Especially comedy spoofs. \"Airplane\", \"The Naked Gun\" trilogy, \"Blazing Saddles\", \"High Anxiety\", and \"Spaceballs\" are some of my favorite comedies that spoof a particular genre. \"Pandemonium\" is not up there with those films. Most of the scenes in this movie had me sitting there in stunned silence because the movie wasn\\'t all that funny. There are a few laughs in the film, but when you watch a comedy, you expect to laugh a lot more than a few times and that\\'s all this film has going for it. Geez, \"Scream\" had more laughs than this film and that was more of a horror film. How bizarre is that?

*1/2 (out of four)'\n", + "Label 0\n", + "Review b\"David Mamet is a very interesting and a very un-equal director. His first movie 'House of Games' was the one I liked best, and it set a series of films with characters whose perspective of life changes as they get into complicated situations, and so does the perspective of the viewer.

So is 'Homicide' which from the title tries to set the mind of the viewer to the usual crime drama. The principal characters are two cops, one Jewish and one Irish who deal with a racially charged area. The murder of an old Jewish shop owner who proves to be an ancient veteran of the Israeli Independence war triggers the Jewish identity in the mind and heart of the Jewish detective.

This is were the flaws of the film are the more obvious. The process of awakening is theatrical and hard to believe, the group of Jewish militants is operatic, and the way the detective eventually walks to the final violent confrontation is pathetic. The end of the film itself is Mamet-like smart, but disappoints from a human emotional perspective.

Joe Mantegna and William Macy give strong performances, but the flaws of the story are too evident to be easily compensated.\"\n", + "Label 0\n", + "Review b'Great documentary about the lives of NY firefighters during the worst terrorist attack of all time.. That reason alone is why this should be a must see collectors item.. What shocked me was not only the attacks, but the\"High Fat Diet\" and physical appearance of some of these firefighters. I think a lot of Doctors would agree with me that,in the physical shape they were in, some of these firefighters would NOT of made it to the 79th floor carrying over 60 lbs of gear. Having said that i now have a greater respect for firefighters and i realize becoming a firefighter is a life altering job. The French have a history of making great documentary\\'s and that is what this is, a Great Documentary.....'\n", + "Label 1\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:44:08.132892: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence\n" + ] + } + ], + "source": [ + "for text_batch, label_batch in raw_train_ds.take(1):\n", + " for i in range(3):\n", + " print(\"Review\", text_batch.numpy()[i])\n", + " print(\"Label\", label_batch.numpy()[i])" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "id": "1e863eb6-4bd7-4da0-b10d-d951b5ee52bd", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Label 0 corresponds to neg\n", + "Label 1 corresponds to pos\n" + ] + } + ], + "source": [ + "print(\"Label 0 corresponds to\", raw_train_ds.class_names[0])\n", + "print(\"Label 1 corresponds to\", raw_train_ds.class_names[1])" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "id": "1593e2e5-df51-4fbf-b4be-c786e740ddab", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Found 25000 files belonging to 2 classes.\n", + "Using 5000 files for validation.\n" + ] + } + ], + "source": [ + "raw_val_ds = tf.keras.utils.text_dataset_from_directory(\n", + " \"aclImdb/train\",\n", + " batch_size=batch_size,\n", + " validation_split=0.2,\n", + " subset=\"validation\",\n", + " seed=seed,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "id": "944fd61d-3926-4296-889a-b2a375a1b039", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Found 25000 files belonging to 2 classes.\n" + ] + } + ], + "source": [ + "raw_test_ds = tf.keras.utils.text_dataset_from_directory(\n", + " \"aclImdb/test\", batch_size=batch_size\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "id": "cb141709-fcc1-4cee-bc98-9c89aaba8648", + "metadata": {}, + "outputs": [], + "source": [ + "def custom_standardization(input_data):\n", + " lowercase = tf.strings.lower(input_data)\n", + " stripped_html = tf.strings.regex_replace(lowercase, \"
\", \" \")\n", + " return tf.strings.regex_replace(\n", + " stripped_html, \"[%s]\" % re.escape(string.punctuation), \"\"\n", + " )" + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "id": "d4e80ea9-536a-4ebc-8b35-1eca73dbba7d", + "metadata": {}, + "outputs": [], + "source": [ + "max_features = 10000\n", + "sequence_length = 250\n", + "\n", + "vectorize_layer = layers.TextVectorization(\n", + " standardize=custom_standardization,\n", + " max_tokens=max_features,\n", + " output_mode=\"int\",\n", + " output_sequence_length=sequence_length,\n", + ")" + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "id": "ad1e5d81-7dae-4b08-b520-ca45501b9510", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "2024-10-03 17:44:10.225130: I tensorflow/core/framework/local_rendezvous.cc:404] Local rendezvous is aborting with status: OUT_OF_RANGE: End of sequence\n" + ] + } + ], + "source": [ + "# Make a text-only dataset (without labels), then call adapt\n", + "train_text = raw_train_ds.map(lambda x, y: x)\n", + "vectorize_layer.adapt(train_text)" + ] + }, + { + "cell_type": "code", + "execution_count": 18, + "id": "80f243f5-edd3-4e1c-bddc-abc1cc6673ef", + "metadata": {}, + "outputs": [], + "source": [ + "def vectorize_text(text, label):\n", + " text = tf.expand_dims(text, -1)\n", + " return vectorize_layer(text), label" + ] + }, + { + "cell_type": "code", + "execution_count": 19, + "id": "8f37e95c-515c-4edb-a1ee-fc47be5df4b9", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Review tf.Tensor(b'Silent Night, Deadly Night 5 is the very last of the series, and like part 4, it\\'s unrelated to the first three except by title and the fact that it\\'s a Christmas-themed horror flick.

Except to the oblivious, there\\'s some obvious things going on here...Mickey Rooney plays a toymaker named Joe Petto and his creepy son\\'s name is Pino. Ring a bell, anyone? Now, a little boy named Derek heard a knock at the door one evening, and opened it to find a present on the doorstep for him. Even though it said \"don\\'t open till Christmas\", he begins to open it anyway but is stopped by his dad, who scolds him and sends him to bed, and opens the gift himself. Inside is a little red ball that sprouts Santa arms and a head, and proceeds to kill dad. Oops, maybe he should have left well-enough alone. Of course Derek is then traumatized by the incident since he watched it from the stairs, but he doesn\\'t grow up to be some killer Santa, he just stops talking.

There\\'s a mysterious stranger lurking around, who seems very interested in the toys that Joe Petto makes. We even see him buying a bunch when Derek\\'s mom takes him to the store to find a gift for him to bring him out of his trauma. And what exactly is this guy doing? Well, we\\'re not sure but he does seem to be taking these toys apart to see what makes them tick. He does keep his landlord from evicting him by promising him to pay him in cash the next day and presents him with a \"Larry the Larvae\" toy for his kid, but of course \"Larry\" is not a good toy and gets out of the box in the car and of course, well, things aren\\'t pretty.

Anyway, eventually what\\'s going on with Joe Petto and Pino is of course revealed, and as with the old story, Pino is not a \"real boy\". Pino is probably even more agitated and naughty because he suffers from \"Kenitalia\" (a smooth plastic crotch) so that could account for his evil ways. And the identity of the lurking stranger is revealed too, and there\\'s even kind of a happy ending of sorts. Whee.

A step up from part 4, but not much of one. Again, Brian Yuzna is involved, and Screaming Mad George, so some decent special effects, but not enough to make this great. A few leftovers from part 4 are hanging around too, like Clint Howard and Neith Hunter, but that doesn\\'t really make any difference. Anyway, I now have seeing the whole series out of my system. Now if I could get some of it out of my brain. 4 out of 5.', shape=(), dtype=string)\n", + "Label neg\n", + "Vectorized review (, )\n" + ] + } + ], + "source": [ + "# retrieve a batch (of 32 reviews and labels) from the dataset\n", + "text_batch, label_batch = next(iter(raw_train_ds))\n", + "first_review, first_label = text_batch[0], label_batch[0]\n", + "print(\"Review\", first_review)\n", + "print(\"Label\", raw_train_ds.class_names[first_label])\n", + "print(\"Vectorized review\", vectorize_text(first_review, first_label))" + ] + }, + { + "cell_type": "code", + "execution_count": 20, + "id": "60c9208a-39ac-4e6c-a603-61038cdf3d10", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "1287 ---> silent\n", + " 313 ---> night\n", + "Vocabulary size: 10000\n" + ] + } + ], + "source": [ + "print(\"1287 ---> \",vectorize_layer.get_vocabulary()[1287])\n", + "print(\" 313 ---> \",vectorize_layer.get_vocabulary()[313])\n", + "print('Vocabulary size: {}'.format(len(vectorize_layer.get_vocabulary())))" + ] + }, + { + "cell_type": "code", + "execution_count": 21, + "id": "3cf90d4b-8dae-44b2-b32b-80cb0092c430", + "metadata": {}, + "outputs": [], + "source": [ + "train_ds = raw_train_ds.map(vectorize_text)\n", + "val_ds = raw_val_ds.map(vectorize_text)\n", + "test_ds = raw_test_ds.map(vectorize_text)" + ] + }, + { + "cell_type": "code", + "execution_count": 22, + "id": "115a5aba-8a00-458f-be25-0aae9f55de22", + "metadata": {}, + "outputs": [], + "source": [ + "AUTOTUNE = tf.data.AUTOTUNE\n", + "\n", + "train_ds = train_ds.cache().prefetch(buffer_size=AUTOTUNE)\n", + "val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)\n", + "test_ds = test_ds.cache().prefetch(buffer_size=AUTOTUNE)" + ] + }, + { + "cell_type": "code", + "execution_count": 23, + "id": "d64f4495-102d-4244-9b42-1ba9976a366e", + "metadata": {}, + "outputs": [], + "source": [ + "embedding_dim = 16" + ] + }, + { + "cell_type": "code", + "execution_count": 24, + "id": "3dc95d22-935f-4091-b0ee-da95174eb9a0", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
Model: \"sequential\"\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1mModel: \"sequential\"\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
+       "┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃\n",
+       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
+       "│ embedding (Embedding)           │ ?                      │   0 (unbuilt) │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ dropout (Dropout)               │ ?                      │   0 (unbuilt) │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ global_average_pooling1d        │ ?                      │   0 (unbuilt) │\n",
+       "│ (GlobalAveragePooling1D)        │                        │               │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ dropout_1 (Dropout)             │ ?                      │   0 (unbuilt) │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ dense (Dense)                   │ ?                      │   0 (unbuilt) │\n",
+       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
+       "
\n" + ], + "text/plain": [ + "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n", + "┃\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0m┃\n", + "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n", + "│ embedding (\u001b[38;5;33mEmbedding\u001b[0m) │ ? │ \u001b[38;5;34m0\u001b[0m (unbuilt) │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ dropout (\u001b[38;5;33mDropout\u001b[0m) │ ? │ \u001b[38;5;34m0\u001b[0m (unbuilt) │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ global_average_pooling1d │ ? │ \u001b[38;5;34m0\u001b[0m (unbuilt) │\n", + "│ (\u001b[38;5;33mGlobalAveragePooling1D\u001b[0m) │ │ │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ dropout_1 (\u001b[38;5;33mDropout\u001b[0m) │ ? │ \u001b[38;5;34m0\u001b[0m (unbuilt) │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ dense (\u001b[38;5;33mDense\u001b[0m) │ ? │ \u001b[38;5;34m0\u001b[0m (unbuilt) │\n", + "└─────────────────────────────────┴────────────────────────┴───────────────┘\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Total params: 0 (0.00 B)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Trainable params: 0 (0.00 B)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Non-trainable params: 0 (0.00 B)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "model = tf.keras.Sequential([\n", + " layers.Embedding(max_features, embedding_dim),\n", + " layers.Dropout(0.2),\n", + " layers.GlobalAveragePooling1D(),\n", + " layers.Dropout(0.2),\n", + " layers.Dense(1, activation='sigmoid')])\n", + "\n", + "model.summary()" + ] + }, + { + "cell_type": "code", + "execution_count": 25, + "id": "d9059b93-7666-46db-bf15-517c4c205df9", + "metadata": {}, + "outputs": [], + "source": [ + "model.compile(loss=losses.BinaryCrossentropy(),\n", + " optimizer='adam',\n", + " metrics=[tf.metrics.BinaryAccuracy(threshold=0.5)])" + ] + }, + { + "cell_type": "code", + "execution_count": 26, + "id": "b1d5959f-1bd8-48da-9815-8239599519b2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Epoch 1/10\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "WARNING: All log messages before absl::InitializeLog() is called are written to STDERR\n", + "I0000 00:00:1727977450.773487 1857915 service.cc:146] XLA service 0xac47fb0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:\n", + "I0000 00:00:1727977450.773523 1857915 service.cc:154] StreamExecutor device (0): NVIDIA RTX A6000, Compute Capability 8.6\n", + "2024-10-03 17:44:10.785495: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:268] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.\n", + "2024-10-03 17:44:10.838694: I external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:531] Loaded cuDNN version 8907\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m 88/625\u001b[0m \u001b[32m━━\u001b[0m\u001b[37m━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m0s\u001b[0m 2ms/step - binary_accuracy: 0.5075 - loss: 0.6925 " + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + "I0000 00:00:1727977451.426198 1857915 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m2s\u001b[0m 2ms/step - binary_accuracy: 0.5797 - loss: 0.6821 - val_binary_accuracy: 0.7248 - val_loss: 0.6141\n", + "Epoch 2/10\n", + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 699us/step - binary_accuracy: 0.7588 - loss: 0.5810 - val_binary_accuracy: 0.8092 - val_loss: 0.4989\n", + "Epoch 3/10\n", + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 477us/step - binary_accuracy: 0.8197 - loss: 0.4674 - val_binary_accuracy: 0.8282 - val_loss: 0.4282\n", + "Epoch 4/10\n", + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 449us/step - binary_accuracy: 0.8514 - loss: 0.3946 - val_binary_accuracy: 0.8402 - val_loss: 0.3870\n", + "Epoch 5/10\n", + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 408us/step - binary_accuracy: 0.8673 - loss: 0.3488 - val_binary_accuracy: 0.8494 - val_loss: 0.3608\n", + "Epoch 6/10\n", + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 406us/step - binary_accuracy: 0.8809 - loss: 0.3174 - val_binary_accuracy: 0.8498 - val_loss: 0.3477\n", + "Epoch 7/10\n", + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 466us/step - binary_accuracy: 0.8899 - loss: 0.2913 - val_binary_accuracy: 0.8554 - val_loss: 0.3325\n", + "Epoch 8/10\n", + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 410us/step - binary_accuracy: 0.8977 - loss: 0.2703 - val_binary_accuracy: 0.8580 - val_loss: 0.3232\n", + "Epoch 9/10\n", + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 397us/step - binary_accuracy: 0.9057 - loss: 0.2539 - val_binary_accuracy: 0.8580 - val_loss: 0.3208\n", + "Epoch 10/10\n", + "\u001b[1m625/625\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 784us/step - binary_accuracy: 0.9084 - loss: 0.2405 - val_binary_accuracy: 0.8638 - val_loss: 0.3121\n" + ] + } + ], + "source": [ + "epochs = 10\n", + "history = model.fit(\n", + " train_ds,\n", + " validation_data=val_ds,\n", + " epochs=epochs)" + ] + }, + { + "cell_type": "code", + "execution_count": 27, + "id": "656afe07-354f-4ff2-8e3e-d02bad6c5958", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m 1/782\u001b[0m \u001b[37m━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[1m21s\u001b[0m 28ms/step - binary_accuracy: 0.9062 - loss: 0.2768" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m782/782\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m1s\u001b[0m 705us/step - binary_accuracy: 0.8542 - loss: 0.3343\n", + "Loss: 0.33130890130996704\n", + "Accuracy: 0.856440007686615\n" + ] + } + ], + "source": [ + "loss, accuracy = model.evaluate(test_ds)\n", + "\n", + "print(\"Loss: \", loss)\n", + "print(\"Accuracy: \", accuracy)" + ] + }, + { + "cell_type": "code", + "execution_count": 28, + "id": "a01d0f13-d0b8-4d78-9ddc-ede5ed402446", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "dict_keys(['binary_accuracy', 'loss', 'val_binary_accuracy', 'val_loss'])" + ] + }, + "execution_count": 28, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "history_dict = history.history\n", + "history_dict.keys()" + ] + }, + { + "cell_type": "code", + "execution_count": 29, + "id": "1f7484c3-3cdf-46d5-b95d-80316f0e6240", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAHHCAYAAABDUnkqAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABVv0lEQVR4nO3dd3gU1f7H8fcmIQ2S0JNAQkLvTUAEpKgoWFCkGBSlqHhFqggClyLlAgqooAiIVwFBEcWAIB0EpSlcKSJiAKVJRyShBtjM74/5ZcmSENIn2f28nmcfZmdnZ75LovvhnDPn2AzDMBARERFxER5WFyAiIiKSlRRuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciIiLiUhRuRCzQpUsXIiMjM/TeESNGYLPZsragXObQoUPYbDZmzZqVo9ddv349NpuN9evXO/al9WeVXTVHRkbSpUuXLD1nWsyaNQubzcahQ4dy/NoimaVwI5KEzWZL0yPpl59IZm3evJkRI0Zw/vx5q0sRcQleVhcgkpvMmTPH6fmnn37K6tWrk+2vXLlypq7z0UcfkZCQkKH3Dh06lEGDBmXq+pJ2mflZpdXmzZsZOXIkXbp0oWDBgk6vxcTE4OGhf4eKpIfCjUgSzz77rNPzH3/8kdWrVyfbf6vLly/j7++f5uvky5cvQ/UBeHl54eWl/3RzSmZ+VlnBx8fH0uuL5EX654BIOjVr1oxq1arx888/06RJE/z9/fn3v/8NwDfffMOjjz5KiRIl8PHxoWzZsowePRq73e50jlvHcSSO15g4cSIzZsygbNmy+Pj4UK9ePbZt2+b03pTG3NhsNnr27MmiRYuoVq0aPj4+VK1alRUrViSrf/369dStWxdfX1/Kli3Lhx9+mOZxPBs2bKB9+/aUKlUKHx8fwsPDefXVV7ly5Uqyz1egQAGOHTtG69atKVCgAMWKFaN///7J/i7Onz9Ply5dCAoKomDBgnTu3DlN3TP/+9//sNlszJ49O9lrK1euxGaz8e233wJw+PBhXnnlFSpWrIifnx9FihShffv2aRpPktKYm7TW/Msvv9ClSxfKlCmDr68vISEhPP/88/z999+OY0aMGMGAAQMAKF26tKPrM7G2lMbc/Pnnn7Rv357ChQvj7+/PPffcw9KlS52OSRw/9OWXXzJmzBjCwsLw9fXlgQce4MCBA3f83LczdepUqlatio+PDyVKlKBHjx7JPvv+/ftp27YtISEh+Pr6EhYWRocOHYiNjXUcs3r1au69914KFixIgQIFqFixouO/I5HM0j//RDLg77//5uGHH6ZDhw48++yzBAcHA+YgzAIFCtCvXz8KFCjAd999x/Dhw4mLi2PChAl3PO/nn3/OhQsX+Ne//oXNZmP8+PG0adOGP//8844tCBs3biQ6OppXXnmFgIAA3nvvPdq2bcuRI0coUqQIADt27KBly5aEhoYycuRI7HY7o0aNolixYmn63F999RWXL1+me/fuFClShK1bt/L+++/z119/8dVXXzkda7fbadGiBfXr12fixImsWbOGt99+m7Jly9K9e3cADMPgiSeeYOPGjbz88stUrlyZhQsX0rlz5zvWUrduXcqUKcOXX36Z7Pj58+dTqFAhWrRoAcC2bdvYvHkzHTp0ICwsjEOHDjFt2jSaNWvGb7/9lq5Wt/TUvHr1av7880+6du1KSEgIe/bsYcaMGezZs4cff/wRm81GmzZt2LdvH/PmzePdd9+laNGiALf9mZw6dYqGDRty+fJlevfuTZEiRZg9ezaPP/44CxYs4Mknn3Q6/s0338TDw4P+/fsTGxvL+PHj6dixIz/99FOaP3OiESNGMHLkSJo3b0737t2JiYlh2rRpbNu2jU2bNpEvXz6uXbtGixYtiI+Pp1evXoSEhHDs2DG+/fZbzp8/T1BQEHv27OGxxx6jRo0ajBo1Ch8fHw4cOMCmTZvSXZNIigwRua0ePXoYt/5n0rRpUwMwpk+fnuz4y5cvJ9v3r3/9y/D39zeuXr3q2Ne5c2cjIiLC8fzgwYMGYBQpUsQ4d+6cY/8333xjAMaSJUsc+954441kNQGGt7e3ceDAAce+Xbt2GYDx/vvvO/a1atXK8Pf3N44dO+bYt3//fsPLyyvZOVOS0ucbN26cYbPZjMOHDzt9PsAYNWqU07G1a9c26tSp43i+aNEiAzDGjx/v2Hfjxg2jcePGBmDMnDkz1XoGDx5s5MuXz+nvLD4+3ihYsKDx/PPPp1r3li1bDMD49NNPHfvWrVtnAMa6deucPkvSn1V6ak7puvPmzTMA44cffnDsmzBhggEYBw8eTHZ8RESE0blzZ8fzvn37GoCxYcMGx74LFy4YpUuXNiIjIw273e70WSpXrmzEx8c7jp08ebIBGLt37052raRmzpzpVNPp06cNb29v46GHHnJcwzAMY8qUKQZgfPLJJ4ZhGMaOHTsMwPjqq69ue+53333XAIwzZ86kWoNIRqlbSiQDfHx86Nq1a7L9fn5+ju0LFy5w9uxZGjduzOXLl/n999/veN6oqCgKFSrkeN64cWPA7Ia4k+bNm1O2bFnH8xo1ahAYGOh4r91uZ82aNbRu3ZoSJUo4jitXrhwPP/zwHc8Pzp/v0qVLnD17loYNG2IYBjt27Eh2/Msvv+z0vHHjxk6fZdmyZXh5eTlacgA8PT3p1atXmuqJiori+vXrREdHO/atWrWK8+fPExUVlWLd169f5++//6ZcuXIULFiQ7du3p+laGak56XWvXr3K2bNnueeeewDSfd2k17/77ru59957HfsKFCjASy+9xKFDh/jtt9+cju/atSve3t6O5+n5nUpqzZo1XLt2jb59+zoNcO7WrRuBgYGObrGgoCDA7Bq8fPlyiudKHDT9zTffZPtgbXFPCjciGVCyZEmnL4xEe/bs4cknnyQoKIjAwECKFSvmGIycdLzB7ZQqVcrpeWLQ+eeff9L93sT3J7739OnTXLlyhXLlyiU7LqV9KTly5AhdunShcOHCjnE0TZs2BZJ/Pl9f32RdK0nrAXMsTGhoKAUKFHA6rmLFimmqp2bNmlSqVIn58+c79s2fP5+iRYty//33O/ZduXKF4cOHEx4ejo+PD0WLFqVYsWKcP38+TT+XpNJT87lz5+jTpw/BwcH4+flRrFgxSpcuDaTt9+F210/pWol38B0+fNhpf2Z+p269LiT/nN7e3pQpU8bxeunSpenXrx///e9/KVq0KC1atOCDDz5w+rxRUVE0atSIF198keDgYDp06MCXX36poCNZRmNuRDIg6b/IE50/f56mTZsSGBjIqFGjKFu2LL6+vmzfvp2BAwem6X/cnp6eKe43DCNb35sWdrudBx98kHPnzjFw4EAqVapE/vz5OXbsGF26dEn2+W5XT1aLiopizJgxnD17loCAABYvXszTTz/tdEdZr169mDlzJn379qVBgwYEBQVhs9no0KFDtn6hPvXUU2zevJkBAwZQq1YtChQoQEJCAi1btsyxL/Ls/r1Iydtvv02XLl345ptvWLVqFb1792bcuHH8+OOPhIWF4efnxw8//MC6detYunQpK1asYP78+dx///2sWrUqx353xHUp3IhkkfXr1/P3338THR1NkyZNHPsPHjxoYVU3FS9eHF9f3xTvlEnL3TO7d+9m3759zJ49m06dOjn2r169OsM1RUREsHbtWi5evOjUEhITE5Pmc0RFRTFy5Ei+/vprgoODiYuLo0OHDk7HLFiwgM6dO/P222879l29ejVDk+alteZ//vmHtWvXMnLkSIYPH+7Yv3///mTnTM+M0xERESn+/SR2e0ZERKT5XOmReN6YmBjKlCnj2H/t2jUOHjxI8+bNnY6vXr061atXZ+jQoWzevJlGjRoxffp0/vOf/wDg4eHBAw88wAMPPMA777zD2LFjGTJkCOvWrUt2LpH0UreUSBZJ/Ndm0n8RX7t2jalTp1pVkhNPT0+aN2/OokWLOH78uGP/gQMHWL58eZreD86fzzAMJk+enOGaHnnkEW7cuMG0adMc++x2O++//36az1G5cmWqV6/O/PnzmT9/PqGhoU7hMrH2W1sq3n///WS3pWdlzSn9fQFMmjQp2Tnz588PkKaw9cgjj7B161a2bNni2Hfp0iVmzJhBZGQkVapUSetHSZfmzZvj7e3Ne++95/SZPv74Y2JjY3n00UcBiIuL48aNG07vrV69Oh4eHsTHxwNmd92tatWqBeA4RiQz1HIjkkUaNmxIoUKF6Ny5M71798ZmszFnzpxsbf5PrxEjRrBq1SoaNWpE9+7dsdvtTJkyhWrVqrFz585U31upUiXKli1L//79OXbsGIGBgXz99dfpHruRVKtWrWjUqBGDBg3i0KFDVKlShejo6HSPR4mKimL48OH4+vrywgsvJJvR97HHHmPOnDkEBQVRpUoVtmzZwpo1axy3yGdHzYGBgTRp0oTx48dz/fp1SpYsyapVq1JsyatTpw4AQ4YMoUOHDuTLl49WrVo5Qk9SgwYNYt68eTz88MP07t2bwoULM3v2bA4ePMjXX3+dbbMZFytWjMGDBzNy5EhatmzJ448/TkxMDFOnTqVevXqOsWXfffcdPXv2pH379lSoUIEbN24wZ84cPD09adu2LQCjRo3ihx9+4NFHHyUiIoLTp08zdepUwsLCnAZKi2SUwo1IFilSpAjffvstr732GkOHDqVQoUI8++yzPPDAA475VqxWp04dli9fTv/+/Rk2bBjh4eGMGjWKvXv33vFurnz58rFkyRLH+AlfX1+efPJJevbsSc2aNTNUj4eHB4sXL6Zv377MnTsXm83G448/zttvv03t2rXTfJ6oqCiGDh3K5cuXne6SSjR58mQ8PT357LPPuHr1Ko0aNWLNmjUZ+rmkp+bPP/+cXr168cEHH2AYBg899BDLly93ulsNoF69eowePZrp06ezYsUKEhISOHjwYIrhJjg4mM2bNzNw4EDef/99rl69So0aNViyZImj9SS7jBgxgmLFijFlyhReffVVChcuzEsvvcTYsWMd8zDVrFmTFi1asGTJEo4dO4a/vz81a9Zk+fLljjvFHn/8cQ4dOsQnn3zC2bNnKVq0KE2bNmXkyJGOu61EMsNm5KZ/VoqIJVq3bs2ePXtSHA8iIpLXaMyNiJu5damE/fv3s2zZMpo1a2ZNQSIiWUwtNyJuJjQ01LHe0eHDh5k2bRrx8fHs2LGD8uXLW12eiEimacyNiJtp2bIl8+bN4+TJk/j4+NCgQQPGjh2rYCMiLkMtNyIiIuJSNOZGREREXIrCjYiIiLgUtxtzk5CQwPHjxwkICEjXlOciIiJiHcMwuHDhAiVKlLjjZJVuF26OHz9OeHi41WWIiIhIBhw9epSwsLBUj3G7cBMQEACYfzmBgYEWVyMiIiJpERcXR3h4uON7PDVuF24Su6ICAwMVbkRERPKYtAwp0YBiERERcSkKNyIiIuJSFG5ERETEpbjdmBsREcladrud69evW12GuABvb+873uadFgo3IiKSIYZhcPLkSc6fP291KeIiPDw8KF26NN7e3pk6j8KNiIhkSGKwKV68OP7+/poYVTIlcZLdEydOUKpUqUz9PinciIhIutntdkewKVKkiNXliIsoVqwYx48f58aNG+TLly/D59GAYhERSbfEMTb+/v4WVyKuJLE7ym63Z+o8CjciIpJh6oqSrJRVv0/qlsoidjts2AAnTkBoKDRuDJ6eVlclIiLiftRykwWioyEyEu67D555xvwzMtLcLyIiri8yMpJJkyal+fj169djs9my/U6zWbNmUbBgwWy9Rm6kcJNJ0dHQrh389Zfz/mPHzP0KOCIiqbPbYf16mDfP/DOTwy1SZbPZUn2MGDEiQ+fdtm0bL730UpqPb9iwISdOnCAoKChD15PUqVsqE+x26NMHDCP5a4YBNhv07QtPPKEuKhGRlERHm/8fTfoPxLAwmDwZ2rTJ+uudOHHCsT1//nyGDx9OTEyMY1+BAgUc24ZhYLfb8fK681dlsWLF0lWHt7c3ISEh6XqPpJ1abjJhw4bkLTZJGQYcPWoeJyIizqxo+Q4JCXE8goKCsNlsjue///47AQEBLF++nDp16uDj48PGjRv5448/eOKJJwgODqZAgQLUq1ePNWvWOJ331m4pm83Gf//7X5588kn8/f0pX748ixcvdrx+a7dUYvfRypUrqVy5MgUKFKBly5ZOYezGjRv07t2bggULUqRIEQYOHEjnzp1p3bp1uv4Opk2bRtmyZfH29qZixYrMmTPH8ZphGIwYMYJSpUrh4+NDiRIl6N27t+P1qVOnUr58eXx9fQkODqZdu3bpunZOUbjJhCS/c1lynIiIu7hTyzeYLd/Z2UV1O4MGDeLNN99k79691KhRg4sXL/LII4+wdu1aduzYQcuWLWnVqhVHjhxJ9TwjR47kqaee4pdffuGRRx6hY8eOnDt37rbHX758mYkTJzJnzhx++OEHjhw5Qv/+/R2vv/XWW3z22WfMnDmTTZs2ERcXx6JFi9L12RYuXEifPn147bXX+PXXX/nXv/5F165dWbduHQBff/017777Lh9++CH79+9n0aJFVK9eHYD//e9/9O7dm1GjRhETE8OKFSto0qRJuq6fYww3ExsbawBGbGxsps+1bp1hmP8Zpv5Yty7TlxIRyVWuXLli/Pbbb8aVK1cy9P7c8P/PmTNnGkFBQUlqWmcAxqJFi+743qpVqxrvv/++43lERITx7rvvOp4DxtChQx3PL168aADG8uXLna71zz//OGoBjAMHDjje88EHHxjBwcGO58HBwcaECRMcz2/cuGGUKlXKeOKJJ9L8GRs2bGh069bN6Zj27dsbjzzyiGEYhvH2228bFSpUMK5du5bsXF9//bURGBhoxMXF3fZ6mZXa71V6vr/VcpMJjRubfcO3uy3fZoPwcPM4ERG5KTe3fNetW9fp+cWLF+nfvz+VK1emYMGCFChQgL17996x5aZGjRqO7fz58xMYGMjp06dve7y/vz9ly5Z1PA8NDXUcHxsby6lTp7j77rsdr3t6elKnTp10fba9e/fSqFEjp32NGjVi7969ALRv354rV65QpkwZunXrxsKFC7lx4wYADz74IBEREZQpU4bnnnuOzz77jMuXL6fr+jlF4SYTPD3NQW+QPOAkPp80SYOJRURuFRqatcdlpfz58zs979+/PwsXLmTs2LFs2LCBnTt3Ur16da5du5bqeW5dPsBms5GQkJCu442U+u2yUXh4ODExMUydOhU/Pz9eeeUVmjRpwvXr1wkICGD79u3MmzeP0NBQhg8fTs2aNXPlwqkKN5nUpg0sWAAlSzrvDwsz92fHaH8RkbwuL7V8b9q0iS5duvDkk09SvXp1QkJCOHToUI7WEBQURHBwMNu2bXPss9vtbN++PV3nqVy5Mps2bXLat2nTJqpUqeJ47ufnR6tWrXjvvfdYv349W7ZsYffu3QB4eXnRvHlzxo8fzy+//MKhQ4f47rvvMvHJsoduBc8CbdqYt3trhmIRkbRJbPlu184MMkkbKHJby3f58uWJjo6mVatW2Gw2hg0blmoLTHbp1asX48aNo1y5clSqVIn333+ff/75J11LFgwYMICnnnqK2rVr07x5c5YsWUJ0dLTj7q9Zs2Zht9upX78+/v7+zJ07Fz8/PyIiIvj222/5888/adKkCYUKFWLZsmUkJCRQsWLF7PrIGaZwk0U8PaFZM6urEBHJOxJbvlOa52bSpNzT8v3OO+/w/PPP07BhQ4oWLcrAgQOJi4vL8ToGDhzIyZMn6dSpE56enrz00ku0aNECz3QkwNatWzN58mQmTpxInz59KF26NDNnzqTZ/3+BFSxYkDfffJN+/fpht9upXr06S5YsoUiRIhQsWJDo6GhGjBjB1atXKV++PPPmzaNq1arZ9IkzzmbkdIeexeLi4ggKCiI2NpbAwECryxERyZOuXr3KwYMHKV26NL6+vpk6l9bmy5iEhAQqV67MU089xejRo60uJ0uk9nuVnu9vtdyIiIil1PKdNocPH2bVqlU0bdqU+Ph4pkyZwsGDB3nmmWesLi3X0YBiERGRPMDDw4NZs2ZRr149GjVqxO7du1mzZg2VK1e2urRcRy03IiIieUB4eHiyO50kZWq5EREREZeicCMiIiIuReFGREREXIrCjYiIiLgUhRsRERFxKQo3IiIi4lIUbkRERNKpWbNm9O3b1/E8MjKSSZMmpfoem83GokWLMn3trDpPakaMGEGtWrWy9RrZSeFGRETcRqtWrWjZsmWKr23YsAGbzcYvv/yS7vNu27aNl156KbPlObldwDhx4gQPP/xwll7L1SjciIiI23jhhRdYvXo1fyVdqfP/zZw5k7p161KjRo10n7dYsWL4+/tnRYl3FBISgo+PT45cK69SuBEREbfx2GOPUaxYMWbNmuW0/+LFi3z11Ve88MIL/P333zz99NOULFkSf39/qlevzrx581I9763dUvv376dJkyb4+vpSpUoVVq9enew9AwcOpEKFCvj7+1OmTBmGDRvG9evXAZg1axYjR45k165d2Gw2bDabo+Zbu6V2797N/fffj5+fH0WKFOGll17i4sWLjte7dOlC69atmThxIqGhoRQpUoQePXo4rpUWCQkJjBo1irCwMHx8fKhVqxYrVqxwvH7t2jV69uxJaGgovr6+REREMG7cOAAMw2DEiBGUKlUKHx8fSpQoQe/evdN87YzQ8gsiIpIlDAMuX7bm2v7+YLPd+TgvLy86derErFmzGDJkCLb/f9NXX32F3W7n6aef5uLFi9SpU4eBAwcSGBjI0qVLee655yhbtix33333Ha+RkJBAmzZtCA4O5qeffiI2NtZpfE6igIAAZs2aRYkSJdi9ezfdunUjICCA119/naioKH799VdWrFjBmjVrAAgKCkp2jkuXLtGiRQsaNGjAtm3bOH36NC+++CI9e/Z0CnDr1q0jNDSUdevWceDAAaKioqhVqxbdunW7818aMHnyZN5++20+/PBDateuzSeffMLjjz/Onj17KF++PO+99x6LFy/myy+/pFSpUhw9epSjR48C8PXXX/Puu+/yxRdfULVqVU6ePMmuXbvSdN0MM9xMbGysARixsbFWlyIikmdduXLF+O2334wrV6449l28aBhmxMn5x8WLaa997969BmCsW7fOsa9x48bGs88+e9v3PProo8Zrr73meN60aVOjT58+jucRERHGu+++axiGYaxcudLw8vIyjh075nh9+fLlBmAsXLjwtteYMGGCUadOHcfzN954w6hZs2ay45KeZ8aMGUahQoWMi0n+ApYuXWp4eHgYJ0+eNAzDMDp37mxEREQYN27ccBzTvn17Iyoq6ra13HrtEiVKGGPGjHE6pl69esYrr7xiGIZh9OrVy7j//vuNhISEZOd6++23jQoVKhjXrl277fUSpfR7lSg939/qlhIREbdSqVIlGjZsyCeffALAgQMH2LBhAy+88AIAdrud0aNHU716dQoXLkyBAgVYuXIlR44cSdP59+7dS3h4OCVKlHDsa9CgQbLj5s+fT6NGjQgJCaFAgQIMHTo0zddIeq2aNWuSP39+x75GjRqRkJBATEyMY1/VqlXx9PR0PA8NDeX06dNpukZcXBzHjx+nUaNGTvsbNWrE3r17AbPra+fOnVSsWJHevXuzatUqx3Ht27fnypUrlClThm7durFw4UJu3LiRrs+ZXgo3IiKSJfz94eJFax7pHcv7wgsv8PXXX3PhwgVmzpxJ2bJladq0KQATJkxg8uTJDBw4kHXr1rFz505atGjBtWvXsuzvasuWLXTs2JFHHnmEb7/9lh07djBkyJAsvUZS+fLlc3pus9lISEjIsvPfddddHDx4kNGjR3PlyhWeeuop2rVrB5irmcfExDB16lT8/Px45ZVXaNKkSbrG/KSXxtyIiEiWsNkgSQNCrvbUU0/Rp08fPv/8cz799FO6d+/uGH+zadMmnnjiCZ599lnAHEOzb98+qlSpkqZzV65cmaNHj3LixAlCQ0MB+PHHH52O2bx5MxEREQwZMsSx7/Dhw07HeHt7Y7fb73itWbNmcenSJUfrzaZNm/Dw8KBixYppqvdOAgMDKVGiBJs2bXIEwMTrJB2DFBgYSFRUFFFRUbRr146WLVty7tw5ChcujJ+fH61ataJVq1b06NGDSpUqsXv3bu66664sqfFWCjciIuJ2ChQoQFRUFIMHDyYuLo4uXbo4XitfvjwLFixg8+bNFCpUiHfeeYdTp06lOdw0b96cChUq0LlzZyZMmEBcXJxTiEm8xpEjR/jiiy+oV68eS5cuZeHChU7HREZGcvDgQXbu3ElYWBgBAQHJbgHv2LEjb7zxBp07d2bEiBGcOXOGXr168dxzzxEcHJyxv5wUDBgwgDfeeIOyZctSq1YtZs6cyc6dO/nss88AeOeddwgNDaV27dp4eHjw1VdfERISQsGCBZk1axZ2u5369evj7+/P3Llz8fPzIyIiIsvqu5W6pURExC298MIL/PPPP7Ro0cJpfMzQoUO56667aNGiBc2aNSMkJITWrVun+bweHh4sXLiQK1eucPfdd/Piiy8yZswYp2Mef/xxXn31VXr27EmtWrXYvHkzw4YNczqmbdu2tGzZkvvuu49ixYqleDu6v78/K1eu5Ny5c9SrV4927drxwAMPMGXKlPT9ZdxB79696devH6+99hrVq1dnxYoVLF68mPLlywPmnV/jx4+nbt261KtXj0OHDrFs2TI8PDwoWLAgH330EY0aNaJGjRqsWbOGJUuWUKRIkSytMSmbYRhGtp09F4qLiyMoKIjY2FgCAwOtLkdEJE+6evUqBw8epHTp0vj6+lpdjriI1H6v0vP9rZYbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERkQxzs3tSJJtl1e+Two2IiKRb4oy3l61aKVNcUuIMzUmXisgITeInIiLp5unpScGCBR3rE/n7+ztm+BXJiISEBM6cOYO/vz9eXpmLJwo3IiKSISEhIQBpXoBR5E48PDwoVapUpoOywk0Wu3EDMhk4RUTyBJvNRmhoKMWLF8/WRRDFfXh7e+PhkfkRM/oaziIxMfDaa1C2LEyebHU1IiI5x9PTM9NjJESykgYUZ5GjR2HpUpg6FQ4csLoaERER96Vwk0WaN4eWLc1uqX//2+pqRERE3JfCTRYaPx48POCrr+DHH62uRkRExD0p3GSh6tWhSxdzu39/0NxWIiIiOU/hJouNGgV+frBpEyxaZHU1IiIi7kfhJouVLGneNQUwcCDo7kgREZGcpXCTDQYMgGLFYP9+mDHD6mpERETci+Xh5oMPPiAyMhJfX1/q16/P1q1bUz3+/Pnz9OjRg9DQUHx8fKhQoQLLli3LoWrTJjAQRowwt0eOhLg4S8sRERFxK5aGm/nz59OvXz/eeOMNtm/fTs2aNWnRosVtp/K+du0aDz74IIcOHWLBggXExMTw0UcfUbJkyRyu/M66dYMKFeDMGfMuKhEREckZNsPC9err169PvXr1mDJlCmAumhUeHk6vXr0YNGhQsuOnT5/OhAkT+P333x0r0qZXXFwcQUFBxMbGEhgYmKn672TRInjySXOA8f795ngcERERSb/0fH9b1nJz7do1fv75Z5o3b36zGA8PmjdvzpYtW1J8z+LFi2nQoAE9evQgODiYatWqMXbsWOx2+22vEx8fT1xcnNMjpzzxBNx7L1y5AsOG5dhlRURE3Jpl4ebs2bPY7XaCg4Od9gcHB3Py5MkU3/Pnn3+yYMEC7HY7y5YtY9iwYbz99tv85z//ue11xo0bR1BQkOMRHh6epZ8jNTYbTJxobs+aBb/8kmOXFhERcVuWDyhOj4SEBIoXL86MGTOoU6cOUVFRDBkyhOnTp9/2PYMHDyY2NtbxOHr0aA5WDPXrQ/v25oR+r7+eo5cWERFxS5aFm6JFi+Lp6cmpU6ec9p86dYqQkJAU3xMaGkqFChWcVp+tXLkyJ0+e5Nq1aym+x8fHh8DAQKdHThs3DvLlg5UrYfXqHL+8iIiIW7Es3Hh7e1OnTh3Wrl3r2JeQkMDatWtp0KBBiu9p1KgRBw4cICEhwbFv3759hIaG4u3tne01Z1TZsvDKK+b2gAGQyhAhERERySRLu6X69evHRx99xOzZs9m7dy/du3fn0qVLdO3aFYBOnToxePBgx/Hdu3fn3Llz9OnTh3379rF06VLGjh1Ljx49rPoIaTZsGAQFwa5d8NlnVlcjIiLiurysvHhUVBRnzpxh+PDhnDx5klq1arFixQrHIOMjR47g4XEzf4WHh7Ny5UpeffVVatSoQcmSJenTpw8DBw606iOkWZEi8O9/m0syDB1qjsPx87O6KhEREddj6Tw3VsjJeW5udfUqVKwIR46Y43BSmMpHREREUpAn5rlxR76+MGaMuT1unDl7sYiIiGQthZsc9swzULu2ud7U6NFWVyMiIuJ6FG5ymIcHTJhgbk+bZi7LICIiIllH4cYCDzwADz8MN25AkpvBREREJAso3Fhk/HizFefrr+E2S2mJiIhIBijcWKRaNfj/6Xzo399cnkFEREQyT+HGQqNGgb8/bN4MCxdaXY2IiIhrULixUIkS8Npr5vagQXD9urX1iIiIuAKFG4sNGADFi5t3TX34odXViIiI5H0KNxYLCIARI8ztkSMhNtbSckRERPI8hZtc4MUXzWUZzp6Ft96yuhoREZG8TeEmF8iX72aoefddOHrU2npERETyMoWbXOLxx6FxY3NxzeHDra5GREQk71K4ySVsNpg40dyePRt27bK2HhERkbxK4SYXuftuiIoyJ/R7/XWrqxEREcmbFG5ymbFjzTE4q1aZDxEREUkfhZtcpkwZ6NHD3B4wAOx2a+sRERHJaxRucqGhQyEoCH75BebMsboaERGRvEXhJhcqUgSGDDG3hw6Fy5etrUdERCQvUbjJpXr1gogIOHYMJk+2uhoREZG8Q+Eml/L1hTFjzO1x4+DMGWvrERERySsUbnKxp5+Gu+6CCxdg1CirqxEREckbFG5yMQ8PmDDB3J4+Hfbts7YeERGRvEDhJpe7/3545BG4cQMGD7a6GhERkdxP4SYPGD/ebMWJjoZNm6yuRkREJHdTuMkDqlaF5583twcMMJdnEBERkZQp3OQRo0aBvz9s2WK24IiIiEjKFG7yiNBQ6N/f3B40CK5ds7YeERGR3ErhJg/p3x+KF4cDB+DDD7PnGnY7rF8P8+aZf2ptKxERyWsUbvKQgAAYOdLcHjkSYmOz9vzR0RAZCffdB888Y/4ZGaluMBERyVsUbvKYF1+ESpXg77/hzTez7rzR0dCuHfz1l/P+Y8fM/Qo4IiKSVyjc5DFeXvDWW+b2pElw9Gjmz2m3Q58+Kd+Flbivb191UYmISN6gcJMHtWoFTZrA1avmquGZtWFD8habpAzDDFEbNmT+WiIiItlN4SYPstlg4kRze84c2Lkzc+c7cSJrjxMREbGSwk0eVa8edOhgtqq8/nrmzhUamrXHiYiIWEnhJg8bMwby5YPVq2Hlyoyfp3FjCAszW4RSYrNBeLh5nIiISG6ncJOHlSkDPXua2wMGZHzAr6cnTJ5sbt8acBKfT5pkHiciIpLbKdzkcUOHQsGCsHs3fPppxs/Tpg0sWAAlSzrvDwsz97dpk6kyRUREcozNMNxrGca4uDiCgoKIjY0lMDDQ6nKyxMSJZstNiRKwf7+5BlVG2e3mXVEnTphjbBo3VouNiIhYLz3f32q5cQE9e0JEBBw/Du++m7lzeXpCs2bw9NPmnwo2IiKS1yjcuABfXxg71tx+6y04fdraekRERKykcOMiOnSAOnXgwgUYNcrqakRERKyjcOMiPDxgwgRz+8MPYd8+a+sRERGxisKNC7nvPnj0UbhxAwYNsroaERERayjcuJjx481WnIULYeNGq6sRERHJeQo3LqZKFXjhBXN7wICUV/oWERFxZQo3LmjkSMifH3780ZyAT0RExJ0o3Lig0FDo39/cHjwYrl2zth4REZGcpHDjovr3h+Bg+OMPmD7d6mpERERyjsKNiypQwOyeAnPem/PnLS1HREQkxyjcuLAXXoDKleHvv+HNN62uRkREJGco3LgwLy9zOQaASZPgyBFLyxEREckRCjcu7rHHoGlTiI+HoUOtrkZERCT7Kdy4OJsNJk40t+fOhR07rK1HREQkuyncuIG6deHpp80J/TSxn4iIuDqFGzcxZgx4e8PatbBypdXViIiIZB+FGzdRujT07GluDxgAdru19YiIiGQXhRs3MmQIFCwIv/4Ks2dbXY2IiEj2ULhxI4UL37xjatgwuHTJ2npERESyg8KNm+nZEyIj4fhxePddq6sRERHJego3bsbHB8aONbffegtOnbK2HhERkaymcOOGoqLM28MvXry5/pSIiIirULhxQx4eMGGCuT1jBsTEWFuPiIhIVlK4cVPNmplLM9jtMGiQ1dWIiIhkHYUbN/bWW2YrzqJFsGGD1dWIiIhkDYUbN1alCrz4ormtZRlERMRVKNy4uZEjIX9++Okn+Oorq6sRERHJPIUbNxcSYrbaAAweDPHx1tYjIiKSWbki3HzwwQdERkbi6+tL/fr12bp1622PnTVrFjabzenh6+ubg9W6ntdeM0POn3/CtGlWVyMiIpI5loeb+fPn069fP9544w22b99OzZo1adGiBadPn77tewIDAzlx4oTjcfjw4Rys2PUUKHBzvpvRo+H8eUvLERERyRTLw80777xDt27d6Nq1K1WqVGH69On4+/vzySef3PY9NpuNkJAQxyM4ODgHK3ZNzz8PlSvDuXMwbpzV1YiIiGScpeHm2rVr/PzzzzRv3tyxz8PDg+bNm7Nly5bbvu/ixYtEREQQHh7OE088wZ49e3KiXJfm5QXjx5vbkyeDGsNERCSvsjTcnD17FrvdnqzlJTg4mJMnT6b4nooVK/LJJ5/wzTffMHfuXBISEmjYsCF//fVXisfHx8cTFxfn9JCUPfqoOblffLw5uFhERCQvsrxbKr0aNGhAp06dqFWrFk2bNiU6OppixYrx4Ycfpnj8uHHjCAoKcjzCw8NzuOK8w2aDiRPN7XnztGq4iIjkTZaGm6JFi+Lp6cmpW5amPnXqFCEhIWk6R758+ahduzYHDhxI8fXBgwcTGxvreBw9ejTTdbuyOnXgzTfN7X794IsvrK1HREQkvSwNN97e3tSpU4e1a9c69iUkJLB27VoaNGiQpnPY7XZ2795NaGhoiq/7+PgQGBjo9JDUvf469OplbnfqBEl+PCIiIrme5d1S/fr146OPPmL27Nns3buX7t27c+nSJbp27QpAp06dGJxkAMioUaNYtWoVf/75J9u3b+fZZ5/l8OHDvJi4joBkms1mdkm1awfXr8OTT8LOnVZXJSIikjZeVhcQFRXFmTNnGD58OCdPnqRWrVqsWLHCMcj4yJEjeHjczGD//PMP3bp14+TJkxQqVIg6deqwefNmqlSpYtVHcEmenjBnDpw5A99/Dw8/DFu2QGSk1ZWJiIikzmYY7rVcYlxcHEFBQcTGxqqLKg3On4fGjeHXX6FiRdi4EYoWtboqERFxN+n5/ra8W0pyt4IFYcUKCA+HmBho1QouX7a6KhERkdtTuJE7KlnSDDiFCsGPP0JUFNy4YXVVIiIiKVO4kTSpUgWWLAFfX/j2W+jeHdyrQ1NERPIKhRtJs0aNzMn9PDzgv/+9udimiIhIbqJwI+nSujV88IG5PXIk3GZiaBEREcso3Ei6vfwyDB1qbr/yCnzzjbX1iIiIJKVwIxkyahQ8/zwkJECHDrB5s9UViYiImBRuJENsNrNL6tFH4epV8xbxvXutrkpEREThRjLBywvmz4e774Zz56BlSzh+3OqqRETE3SncSKbkzw9Ll0KFCnDkiLlMQ2ys1VWJiIg7U7iRTCta1JzkLyQEfvnFvKMqPt7qqkRExF0p3EiWKF0ali2DgABYvx46dTIHG4uIiOQ0hRvJMrVrQ3Q05MsHX34J/fppFmMREcl5CjeSpZo3h9mzze3Jk2HiRGvrERER96NwI1nu6advhprXX4e5c62tR0RE3IvCjWSL116DV181t7t2hVWrrK1HRETch8KNZJuJE83Zi2/cgLZtYft2qysSERF3oHAj2cbDA2bNgvvvh4sXzTlw/vzT6qpERMTVKdxItvLxgYULoWZNOH0aWrQw/xQREckuCjeS7QIDYflyiIiAAwfgscfMlhwREZHskKFwc/ToUf766y/H861bt9K3b19mzJiRZYWJawkNhZUroUgR2LYNnnoKrl+3uioREXFFGQo3zzzzDOvWrQPg5MmTPPjgg2zdupUhQ4YwatSoLC1QXEfFivDtt+DnZ7bkvPSSJvkTEZGsl6Fw8+uvv3L33XcD8OWXX1KtWjU2b97MZ599xqxZs7KyPnEx99xjriSeONh42DCrKxIREVeToXBz/fp1fHx8AFizZg2PP/44AJUqVeLEiRNZV524pFat4MMPze0xY2DqVGvrERER15KhcFO1alWmT5/Ohg0bWL16NS1btgTg+PHjFClSJEsLFNf04oswcqS53bOnuSaViIhIVshQuHnrrbf48MMPadasGU8//TQ1a9YEYPHixY7uKpE7GTbs5ribZ56BDRusrkhERFyBzTAyNqTTbrcTFxdHoUKFHPsOHTqEv78/xYsXz7ICs1pcXBxBQUHExsYSGBhodTlu78YNaNcOvvkGChaEjRuhalWrqxIRkdwmPd/fGWq5uXLlCvHx8Y5gc/jwYSZNmkRMTEyuDjaS+3h5wbx50LAhnD8PLVvC0aNWVyUiInlZhsLNE088waeffgrA+fPnqV+/Pm+//TatW7dm2rRpWVqguD4/P1iyBCpVgr/+Mpdp+Ocfq6sSEZG8KkPhZvv27TRu3BiABQsWEBwczOHDh/n000957733srRAcQ+FC8OKFVCiBOzZA61bw9WrVlclIiJ5UYbCzeXLlwkICABg1apVtGnTBg8PD+655x4OHz6cpQWK+4iIMCf3CwyEH36AZ58Fuz1j57LbYf16s8tr/fqMn0dERPKeDIWbcuXKsWjRIo4ePcrKlSt56KGHADh9+rQG6Uqm1KhhDi729oavv4Y+fdI/i3F0NERGwn33mXdh3Xef+Vy3m4uIuIcMhZvhw4fTv39/IiMjufvuu2nQoAFgtuLUrl07SwsU99OsGcyZAzYbfPABvPlm2t8bHW3efZVk6TMAjh0z9yvgiIi4vgzfCn7y5ElOnDhBzZo18fAwM9LWrVsJDAykUqVKWVpkVtKt4HnHe++ZLTdgLtXQuXPqx9vtZgvNrcEmkc0GYWFw8CB4emZlpSIikt3S8/2d4XCTKHF18LCwsMycJsco3OQtAwfC+PFmGFmyxLyT6nbWrze7oO5k3TqzdUhERPKObJ/nJiEhgVGjRhEUFERERAQREREULFiQ0aNHk5CQkKGiRVIybtzNgcXt2sG2bbc/Nq3Lmmn5MxER1+aVkTcNGTKEjz/+mDfffJNGjRoBsHHjRkaMGMHVq1cZM2ZMlhYp7svDAz7+GE6fhlWr4NFHYfNmKFcu+bGhoWk7Z1qPExGRvClD3VIlSpRg+vTpjtXAE33zzTe88sorHDt2LMsKzGrqlsqbLlwwu5x+/hnKlDEDTnCw8zGJY26OHUv5DiuNuRERybuyvVvq3LlzKQ4arlSpEufOncvIKUVSFRAAS5eawebPP80WnAsXnI/x9ITJk81tm835tcTnkyYp2IiIuLoMhZuaNWsyZcqUZPunTJlCjRo1Ml2USEqCg81ZjIsWNVtw2rWDa9ecj2nTBhYsgJIlnfeHhZn727TJuXpFRMQaGeqW+v7773n00UcpVaqUY46bLVu2cPToUZYtW+ZYmiE3UrdU3rdtm3m30+XL8NxzMHt28pYaux02bDAHD4eGQuPGarEREcnLsr1bqmnTpuzbt48nn3yS8+fPc/78edq0acOePXuYM2dOhooWSat69cxWGE9Pc7K/wYOTH+PpaQagp582/1SwERFxH5me5yapXbt2cdddd2HPxQv5qOXGdcyaBV27mtuTJ0Pv3paWIyIi2SjbW25EcoMuXWDsWHO7b1/48ksrqxERkdxC4UbytEGDoEcP89bv554zZykWERH3pnAjeZrNZnZJtW1r3jnVujXs3m11VSIiYqV0zVDc5g730Z4/fz4ztYhkiKcnzJ1rzmK8YQO0bAlbtkCpUlZXJiIiVkhXuAkKCrrj6506dcpUQSIZ4esL33xj3vK9Z48ZcDZuhMKFra5MRERyWpbeLZUX6G4p1/bXX9Cggflnw4awZg34+VldlYiIZJbulhK3FRZmzmJcsKC5/lS7dqAVQURE3IvCjbicqlVh8WLw8YFly6BKFfj6a6urEhGRnKJwIy6pcWP4/nuoXBlOnTJbcNq0MZdjEBER16ZwIy6rfn3YsQOGDQMvL1i40GzFmTnTnBdHRERck8KNuDQfHxg1ylxFvE4dOH8enn8eHnoIDh60ujoREckOCjfiFmrUgB9/hAkTzNvG16yBatXMCQBz8VJoIiKSAQo34ja8vKB/f3MG46ZN4fJlc02qe++F336zujoREckqCjfidsqVg+++g+nTISDAbNGpXRtGjzaXcBARkbxN4UbckocH/OtfZovNY4+ZoWb4cKhbF7Zts7o6ERHJDIUbcWthYeacOJ9/DkWLml1W99wDAwaY3VYiIpL3KNyI27PZ4OmnzVacZ56BhASYONEchLx+vdXViYhIeinciPy/YsXgs89gyRIoWRL++APuu8/svoqNtbo6ERFJK4UbkVs89pjZivPyy+bzGTPMyf+WLLG2LhERSRuFG5EUBAbCtGlmt1S5cnD8ODz+uNltdeaM1dWJiEhqFG5EUtG0KfzyC7z+unmH1bx55npVn32mJRxERHIrhRuRO/Dzg7fegp9+MgcZ//03PPsstGoFR49aXZ2IiNxK4UYkjerWhf/9D/7zH/D2hqVLoWpVczLAhASrqxMRkUQKNyLpkC8fDBkCO3dCgwZw4QJ0727eVbVvn9XViYgIKNyIZEjlyrBhg7nwpr8//PAD1KwJ48fDjRtWVyci4t5yRbj54IMPiIyMxNfXl/r167N169Y0ve+LL77AZrPRunXr7C1QJAWentC7N+zZAw8+CFevwsCBUL8+7NpldXUiIu7L8nAzf/58+vXrxxtvvMH27dupWbMmLVq04PTp06m+79ChQ/Tv35/GjRvnUKUiKYuMhJUrYeZMKFgQtm83x+cMHWoGHhERyVmWh5t33nmHbt260bVrV6pUqcL06dPx9/fnk08+ue177HY7HTt2ZOTIkZQpUyYHqxVJmc0GXbrA3r3Qtq3ZNTVmjLna+ObNVlcnIuJeLA03165d4+eff6Z58+aOfR4eHjRv3pwtW7bc9n2jRo2iePHivPDCCzlRpkiahYTAggXmIzgYfv8d7r3X7L66eNHq6kRE3IOl4ebs2bPY7XaCg4Od9gcHB3Py5MkU37Nx40Y+/vhjPvroozRdIz4+nri4OKeHSHZr29Zsxena1Zzs7/33oVo1s/tKRESyl+XdUulx4cIFnnvuOT766COKFi2apveMGzeOoKAgxyM8PDybqxQxFSoEn3xiBprISDh8GFq2NLuvzp2zujoREddlabgpWrQonp6enDp1ymn/qVOnCAkJSXb8H3/8waFDh2jVqhVeXl54eXnx6aefsnjxYry8vPjjjz+SvWfw4MHExsY6Hkc1pazksIcegt27oU8fc2zO7NnmQpxff211ZSIirsnScOPt7U2dOnVYu3atY19CQgJr166lQYMGyY6vVKkSu3fvZufOnY7H448/zn333cfOnTtTbJXx8fEhMDDQ6SGS0woUgEmTYNMmc46cU6egXTto0wZOnLC6OhER12J5t1S/fv346KOPmD17Nnv37qV79+5cunSJrl27AtCpUycGDx4MgK+vL9WqVXN6FCxYkICAAKpVq4a3t7eVH0Xkjho0gB07YNgw8PKChQvNVpyZM7UQp4hIVrE83ERFRTFx4kSGDx9OrVq12LlzJytWrHAMMj5y5Agn9E9bcSE+PjBqlLlOVZ06cP48PP+82X118KDV1YmI5H02w3Cvfy/GxcURFBREbGysuqjEcjduwLvvwvDh5oR//v4wdiz07GnOgCwiIqb0fH9b3nIj4s68vGDAAPjlF2jaFC5fhr59zblxfvvN6upERPImhRuRXKB8efjuO5g+HQIC4McfzdmNR46E1ath3jxYvx7sdqsrFRHJ/dQtJZLL/PUXvPwyLF2a/LWwMHMl8jZtcr4uERErqVtKJA8LCzNnNk7JX3+Zsx9HR+dsTSIieYnCjUguY7eb425S07WrOeOxiIgkp3Ajksts2GC20KQmLg7KlDFbcdau1Rw5IiJJKdyI5DJpndYpIcHsnmre3Jz1ePJkc84cERF3p3AjksuEhqbtuE8+gVdeMZd2iIkxu7JKloRu3WDnzuysUEQkd1O4EcllGjc2BxXbbCm/brNBeDh06gQffADHj8PUqVC1qjlPzn//a95G3rAhzJ1rTg4oIuJOFG5EchlPT7OLCZIHnMTnkybdnME4IAC6dzdXHv/hB4iKMicH3LIFnnvODEKDB8OhQzn1CURErKVwI5ILtWkDCxaY3UxJhYWZ+1Oa58ZmM1t9vvgCjh6F0aPN48+ehTffNAcgt2oFy5eb43VERFyVJvETycXsdvPuqRMnzLE4jRunb82pGzfg22/NbqvVq2/uL1PGbO3p2hWKFMn6ukVEslp6vr8VbkTcREyMubzDzJkQG2vu8/GBDh3Mgcn16t1+nI+IiNU0Q7GIJFOxorkC+bFjNwcdx8fD7NlQv74Zbj75xByULCKSlynciLiZ/PnhhRfg559vDjr29jafv/CCOU7ntddg/36rKxURyRiFGxE3ZbPBPffAp5+aMyK/9RZERsI//8A770CFCtCiBXzzjVYjF5G8ReFGRChWDF5/HQ4cMFcjf+QRM/ysWgWtW5sDkMeOhVOnrK5UROTOFG5ExMHT0ww2S5eaQef11827qY4cgSFDzDlznnkGNm7UelYiknsp3IhIisqUMbuq/vrL7Lq65x64fh3mzTNvSa9Z07z76uJFqysVEXGmcCMiqfL1NQcdb9lyc9Cxn585I3L37lCiBPTqBb/9ZnWlIiImhRsRSbO77jJvIz92zLytvHx5uHABpkwx17a67z5zBuXr162uVETcmcKNiKRboULmKuS//27OfPzkk+DhAevXQ/v2EBEBI0aYIUhEJKcp3IhIhnl4QPPmEB1tLsw5dCgEB5vLRYwcaYacdu1g3ToNQBaRnKPlF0QkS127BgsXmutZ/fDDzf2VKsGLL8K995qDkX19ratRRPIerS2VCoUbkZyzezdMmwZz5jjfVZUvH1Svbi75kPioUgW8vKyrVURyN4WbVCjciOS8uDiYO9dcoXzbNjh7Nvkxfn7mgOV69aBuXfPPcuXMri8REYWbVCjciFjLMODwYTPk/O9/N/+8cCH5sUFBN4NO4iMsTKuXi7gjhZtUKNyI5D4JCbBvnxl0Eh87dpirlt8qONg57NSrB0WL5nzNIpKzFG5SoXAjkjdcvw6//uoceH79NeVFPCMjnVt46tQB/ect4loUblKhcCOSd12+DLt2OQeemJjkx9lsULGic+tOrVq6Q0skL1O4SYXCjYhriY01l4VIGniOHEl+nJdX8ju0qlbVHVoieYXCTSoUbkRc3+nTzmFn2zY4cyb5cb6+ULu2c+ApX153aInkRgo3qVC4EXE/hmG25tx6h1ZcXPJjg4LMMTtJA094uO7QErGawk0qFG5EBMw7tPbvT36H1tWryY8tXtx5wPLdd0OxYjlfs4g7U7hJhcKNiNzO9euwZ49z4Nm9O+U7tCpUgMaNzUeTJuYdW2rdEck+CjepULgRyXl2O2zYYC6oGRpqBgJPT6urSpsrV5LfofX778mPK1nyZthp3NgcrKyxOyJZR+EmFQo3IjkrOhr69IG//rq5LywMJk+GNm2sqyszzp2DTZvMwLZhgzl+58YN52MKFTIXCU0MO3fdBd7e1tQr4goUblKhcCOSc6KjoV07c0BvUondNwsW5N2Ak9Tly/DTT2bQ+eEH2LLF3JeUnx/cc8/Nbqx77oH8+a2pVyQvUrhJhcKNSM6w281xKElbbJKy2cwWnIMH804XVVpdv24OTk5s2dm4Ef7+2/kYLy+zNSexZefee6FIEWvqFckLFG5SoXAjkjPWr4f77rvzcevWQbNm2V2NtRISzHE6P/xwM/AcPZr8uKpVncfthIfnfK0iuVV6vr81N6eIZIsTJ7L2uLzMwwOqVDEfL79s7jt8+GbQ+eEHM/zs2WM+pk83j4mIMLuwEsNOxYq6I0skLRRuRCRbhIZm7XGuJiLCfDz7rPn8zBmz+yox8OzYYQagOXPMB5hz6yQOUm7SBGrW1PIRIilRt5SIZIvEMTfHjiUfUAyuPeYmK1y4AD/+eLMr66efkk8wWKAANGx4s2Xn7rvNgcsirkhjblKhcCOScxLvlgLngONqd0vlhPh4c4HQpIOUY2Odj/H2NmdQTgw7jRqZy0mIuAKFm1Qo3IjkrJTmuQkPh0mTFGwyw26HX3+9GXYSJ0lMymaDGjWcx+2EhFhTr0hmKdykQuFGJOfl5RmK8wrDgD//vDlAecMGOHAg+XHlyjnffl6mjH4Wkjco3KRC4UZE3MWJE86DlHftSj7+ydvbDDwVKkD58s5/hoTo7izJPRRuUqFwIyLu6vx52LzZedmI+PjbH1+gwM2gc2v4KVw4x8oWARRuUqVwIyJistvhyBHYvx/27TMfiduHDpmTD95OkSLOgSdxu3x5LSsh2UPhJhUKNyIidxYfb96mnzTwJG4fO5b6e0uUSLm1p0wZ8PHJmfrF9SjcpELhRkQkcy5dMgcr39ras29f8jW0kvLwMOc+Sqmrq1QpDWyW1CncpELhRkQk+5w7Z4adlLq6Ll68/fu8vaFs2ZQHNoeGamCzKNykSuFGRCTnGQacPHkz6CQNP3/8kfrA5vz5bz+wWSupuw+Fm1Qo3IiI5C52u7lKekrjew4eTH1gc+HC5q3sQUHm0hP+/uYjo9uJf6qLLPdRuEmFwo2ISN5x7drNgc1Jw8/+/c6zXmc1b++sCUspBaek2z4+6nJLq/R8f2s9WRERybW8vaFiRfNxq8SBzQcPmtuXL5uPK1fSt534/MqVm+e+ds183Lp+V1az2W4Gnfz5zcVky5Y1H+XK3dwuUkQhKD3UciMiIoLZ/XX1avrDUUbClN2evtqCglIOPWXLQsmS5p1ork4tNyIiIunk4XGz2yi7Bypfv5489Fy4AIcPmwOs//jDbJX64w84ftxsQdq+3XzcysfHnEPo1uBTrhxERJitX+5GLTciImmkBUDFCpcvm4uiJoaepMHn0KHUW4E8PMw5hFIKPmXKmEts5BUaUJwKhRsRyYjoaOjTx3kQa1gYTJ4MbdpYV5e4txs3zCU0EsNO0uDzxx/O44hSEhycPPTk1nE+CjepULgRkfSKjoZ27ZKvqJ34P/4FCxRwJPdJnFsoadhJGn7OnUv9/YGBtw8+VozzUbhJhcKNiKSH3W4uGXC7245tNrMF5+BBdVFJ3vLPP86hJ2nwudP6YUnH+dwafCIjs2ecj8JNKhRuRCQ91q+H++6783Hr1kGzZtldjUjOuHLFDOwpdXcdOmR2h92Oh4c5Hm39+qytSXdLiYhkkRMnsvY4kbzAzw+qVDEft7pxw5xR+nbjfC5ftn71d4UbEZFUhIZm7XEieZ2XF5QubT4efND5NcOAU6fMgGMlhRsRkVQ0bmyOqTl2LPmAYrg55qZx45yvTSS3sdkgJMTqKsAN5jQUEck4T0/zdm9Iflts4vNJkzSYWCQ3UbgREbmDNm3M271LlnTeHxam28BFcqNcEW4++OADIiMj8fX1pX79+mzduvW2x0ZHR1O3bl0KFixI/vz5qVWrFnPmzMnBakXEHbVpY94lsm4dfP65+efBgwo2IrmR5WNu5s+fT79+/Zg+fTr169dn0qRJtGjRgpiYGIoXL57s+MKFCzNkyBAqVaqEt7c33377LV27dqV48eK0aNHCgk8gIu7C01O3e4vkBZbPc1O/fn3q1avHlClTAEhISCA8PJxevXoxaNCgNJ3jrrvu4tFHH2X06NF3PFbz3IiIiOQ96fn+trRb6tq1a/z88880b97csc/Dw4PmzZuzZcuWO77fMAzWrl1LTEwMTZo0yc5SRUREJI+wtFvq7Nmz2O12goODnfYHBwfz+++/3/Z9sbGxlCxZkvj4eDw9PZk6dSoP3nqz/f+Lj48nPj7e8TwuLi5rihcREZFcyfIxNxkREBDAzp07uXjxImvXrqVfv36UKVOGZil0ho8bN46RI0fmfJEiIiJiCUvDTdGiRfH09OTUqVNO+0+dOkVIKrMAeXh4UK5cOQBq1arF3r17GTduXIrhZvDgwfTr18/xPC4ujvDw8Kz5ACIiIpLrWDrmxtvbmzp16rB27VrHvoSEBNauXUuDBg3SfJ6EhASnrqekfHx8CAwMdHqIiIiI67K8W6pfv3507tyZunXrcvfddzNp0iQuXbpE165dAejUqRMlS5Zk3LhxgNnNVLduXcqWLUt8fDzLli1jzpw5TJs2zcqPISIiIrmE5eEmKiqKM2fOMHz4cE6ePEmtWrVYsWKFY5DxkSNH8PC42cB06dIlXnnlFf766y/8/PyoVKkSc+fOJSoqyqqPICKSp9jtsGGDuZJ5aKi5LpaWjxBXYvk8NzlN89yIiDuLjoY+feCvv27uCwsz18/SbMuSm+WZeW5ERCTnREdDu3bOwQbMFc/btTNfF3EFCjciIm7AbjdbbFJqq0/c17eveZxIXqdwIyLiBjZsSN5ik5RhwNGj5nEieZ3CjYiIGzhxImuPE8nNFG5ERNxAaGjWHieSmynciIi4gcaNzbuibLaUX7fZIDzcPE4kr1O4ERFxA56e5u3ekDzgJD6fNEnz3YhrULgREXETbdrAggVQsqTz/rAwc7/muRFXYfkMxSIiknPatIEnntAMxeLaFG5ERNyMpyc0a2Z1FSLZR91SIiIi4lIUbkRERMSlKNyIiIiIS1G4EREREZeiAcUiIpJn2e2680uSU7gREZE8KTraXOk86YKgYWHmZIWas8e9qVtKRETynOhoaNcu+Urnx46Z+6OjralLcgeFGxERyVPsdrPFxjCSv5a4r29f8zhxTwo3IiKSp2zYkLzFJinDgKNHzePEPSnciIhInnLiRNYeJ65H4UZERPKU0NCsPU5cj8KNiIjkKY0bm3dF2Wwpv26zQXi4eZy4J4UbERHJUzw9zdu9IXnASXw+aZLmu3FnCjciIpLntGkDCxZAyZLO+8PCzP2a58a9aRI/ERHJk9q0gSee0AzFkpzCjYiI5FmentCsmdVVSG6jbikRERFxKWq5ERERsZgWAM1aCjciIiIW0gKgWU/dUiIiIhbRAqDZQ+FGRETEAloANPso3IiIiFhAC4BmH4UbERERC2gB0OyjcCMiImIBLQCafRRuRERELKAFQLOPwo2IiIgFtABo9lG4ERERsYgWAM0emsRPRETEQq60AGhumWlZ4UZERMRirrAAaG6aaVndUiIiIpIpuW2mZYUbERERybDcONOywo2IiIhkWG6caVnhRkRERDIsN860rHAjIiIiGZYbZ1pWuBEREZEMy40zLSvciIiISIblxpmWFW5EREQkU3LbTMuaxE9EREQyLTfNtKxwIyIiIlkit8y0rG4pERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FIUbERERcSluN0OxYRgAxMXFWVyJiIiIpFXi93bi93hq3C7cXLhwAYDw8HCLKxEREZH0unDhAkFBQakeYzPSEoFcSEJCAsePHycgIADbrWuzC2Cm4/DwcI4ePUpgYKDV5bg9/TxyF/08ch/9THKX7Pp5GIbBhQsXKFGiBB4eqY+qcbuWGw8PD8LCwqwuI08IDAzU/yhyEf08chf9PHIf/Uxyl+z4edypxSaRBhSLiIiIS1G4EREREZeicCPJ+Pj48MYbb+Dj42N1KYJ+HrmNfh65j34muUtu+Hm43YBiERERcW1quRERERGXonAjIiIiLkXhRkRERFyKwo2IiIi4FIUbcRg3bhz16tUjICCA4sWL07p1a2JiYqwuS4A333wTm81G3759rS7FrR07doxnn32WIkWK4OfnR/Xq1fnf//5ndVluyW63M2zYMEqXLo2fnx9ly5Zl9OjRaVp3SDLvhx9+oFWrVpQoUQKbzcaiRYucXjcMg+HDhxMaGoqfnx/Nmzdn//79OVafwo04fP/99/To0YMff/yR1atXc/36dR566CEuXbpkdWlubdu2bXz44YfUqFHD6lLc2j///EOjRo3Ily8fy5cv57fffuPtt9+mUKFCVpfmlt566y2mTZvGlClT2Lt3L2+99Rbjx4/n/ffft7o0t3Dp0iVq1qzJBx98kOLr48eP57333mP69On89NNP5M+fnxYtWnD16tUcqU+3gsttnTlzhuLFi/P999/TpEkTq8txSxcvXuSuu+5i6tSp/Oc//6FWrVpMmjTJ6rLc0qBBg9i0aRMbNmywuhQBHnvsMYKDg/n4448d+9q2bYufnx9z5861sDL3Y7PZWLhwIa1btwbMVpsSJUrw2muv0b9/fwBiY2MJDg5m1qxZdOjQIdtrUsuN3FZsbCwAhQsXtrgS99WjRw8effRRmjdvbnUpbm/x4sXUrVuX9u3bU7x4cWrXrs1HH31kdVluq2HDhqxdu5Z9+/YBsGvXLjZu3MjDDz9scWVy8OBBTp486fT/raCgIOrXr8+WLVtypAa3WzhT0iYhIYG+ffvSqFEjqlWrZnU5bumLL75g+/btbNu2zepSBPjzzz+ZNm0a/fr149///jfbtm2jd+/eeHt707lzZ6vLczuDBg0iLi6OSpUq4enpid1uZ8yYMXTs2NHq0tzeyZMnAQgODnbaHxwc7HgtuyncSIp69OjBr7/+ysaNG60uxS0dPXqUPn36sHr1anx9fa0uRzADf926dRk7diwAtWvX5tdff2X69OkKNxb48ssv+eyzz/j888+pWrUqO3fupG/fvpQoUUI/D1G3lCTXs2dPvv32W9atW0dYWJjV5biln3/+mdOnT3PXXXfh5eWFl5cX33//Pe+99x5eXl7Y7XarS3Q7oaGhVKlSxWlf5cqVOXLkiEUVubcBAwYwaNAgOnToQPXq1Xnuued49dVXGTdunNWlub2QkBAATp065bT/1KlTjteym8KNOBiGQc+ePVm4cCHfffcdpUuXtrokt/XAAw+we/dudu7c6XjUrVuXjh07snPnTjw9Pa0u0e00atQo2dQI+/btIyIiwqKK3Nvly5fx8HD+CvP09CQhIcGiiiRR6dKlCQkJYe3atY59cXFx/PTTTzRo0CBHalC3lDj06NGDzz//nG+++YaAgABH32hQUBB+fn4WV+deAgICko11yp8/P0WKFNEYKIu8+uqrNGzYkLFjx/LUU0+xdetWZsyYwYwZM6wuzS21atWKMWPGUKpUKapWrcqOHTt45513eP75560uzS1cvHiRAwcOOJ4fPHiQnTt3UrhwYUqVKkXfvn35z3/+Q/ny5SldujTDhg2jRIkSjjuqsp0h8v+AFB8zZ860ujQxDKNp06ZGnz59rC7DrS1ZssSoVq2a4ePjY1SqVMmYMWOG1SW5rbi4OKNPnz5GqVKlDF9fX6NMmTLGkCFDjPj4eKtLcwvr1q1L8fuic+fOhmEYRkJCgjFs2DAjODjY8PHxMR544AEjJiYmx+rTPDciIiLiUjTmRkRERFyKwo2IiIi4FIUbERERcSkKNyIiIuJSFG5ERETEpSjciIiIiEtRuBERERGXonAjIm7JZrOxaNEiq8sQkWygcCMiOa5Lly7YbLZkj5YtW1pdmoi4AK0tJSKWaNmyJTNnznTa5+PjY1E1IuJK1HIjIpbw8fEhJCTE6VGoUCHA7DKaNm0aDz/8MH5+fpQpU4YFCxY4vX/37t3cf//9+Pn5UaRIEV566SUuXrzodMwnn3xC1apV8fHxITQ0lJ49ezq9fvbsWZ588kn8/f0pX748ixcvdrz2zz//0LFjR4oVK4afnx/ly5dPFsZEJHdSuBGRXGnYsGG0bduWXbt20bFjRzp06MDevXsBuHTpEi1atKBQoUJs27aNr776ijVr1jiFl2nTptGjRw9eeukldu/ezeLFiylXrpzTNUaOHMlTTz3FL7/8wiOPPELHjh05d+6c4/q//fYby5cvZ+/evUybNo2iRYvm3F+AiGRcji3RKSLy/zp37mx4enoa+fPnd3qMGTPGMAxzhfqXX37Z6T3169c3unfvbhiGYcyYMcMoVKiQcfHiRcfrS5cuNTw8PIyTJ08ahmEYJUqUMIYMGXLbGgBj6NChjucXL140AGP58uWGYRhGq1atjK5du2bNBxaRHKUxNyJiifvuu49p06Y57StcuLBju0GDBk6vNWjQgJ07dwKwd+9eatasSf78+R2vN2rUiISEBGJiYrDZbBw/fpwHHngg1Rpq1Kjh2M6fPz+BgYGcPn0agO7du9O2bVu2b9/OQw89ROvWrWnYsGGGPquI5CyFGxGxRP78+ZN1E2UVPz+/NB2XL18+p+c2m42EhAQAHn74YQ4fPsyyZctYvXo1DzzwAD169GDixIlZXq+IZC2NuRGRXOnHH39M9rxy5coAVK5cmV27dnHp0iXH65s2bcLDw4OKFSsSEBBAZGQka9euzVQNxYoVo3PnzsydO5dJkyYxY8aMTJ1PRHKGWm5ExBLx8fGcPHnSaZ+Xl5dj0O5XX31F3bp1uffee/nss8/YunUrH3/8MQAdO3bkjTfeoHPnzowYMYIzZ87Qq1cvnnvuOYKDgwEYMWIEL7/8MsWLF+fhhx/mwoULbNq0iV69eqWpvuHDh1OnTh2qVq1KfHw83377rSNciUjupnAjIpZYsWIFoaGhTvsqVqzI77//Dph3Mn3xxRe88sorhIaGMm/ePKpUqQKAv78/K1eupE+fPtSrVw9/f3/atm3LO++84zhX586duXr1Ku+++y79+/enaNGitGvXLs31eXt7M3jwYA4dOoSfnx+NGzfmiy++yIJPLiLZzWYYhmF1ESIiSdlsNhYuXEjr1q2tLkVE8iCNuRERERGXonAjIiIiLkVjbkQk11FvuYhkhlpuRERExKUo3IiIiIhLUbgRERERl6JwIyIiIi5F4UZERERcisKNiIiIuBSFGxEREXEpCjciIiLiUhRuRERExKX8H9OYtlmx9fX1AAAAAElFTkSuQmCC", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "acc = history_dict['binary_accuracy']\n", + "val_acc = history_dict['val_binary_accuracy']\n", + "loss = history_dict['loss']\n", + "val_loss = history_dict['val_loss']\n", + "\n", + "epochs = range(1, len(acc) + 1)\n", + "\n", + "# \"bo\" is for \"blue dot\"\n", + "plt.plot(epochs, loss, 'bo', label='Training loss')\n", + "# b is for \"solid blue line\"\n", + "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n", + "plt.title('Training and validation loss')\n", + "plt.xlabel('Epochs')\n", + "plt.ylabel('Loss')\n", + "plt.legend()\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 30, + "id": "af51178e-fe0b-40ca-9260-2190fb52d960", + "metadata": {}, + "outputs": [ + { + "data": { + "image/png": "iVBORw0KGgoAAAANSUhEUgAAAkAAAAHHCAYAAABXx+fLAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAABZ5klEQVR4nO3deVhUZf8G8HsYZNgRQXYERHNFTFTcSEsLtUjcQjNFs0xzwdBUcs9XKTXDLX3151a5kIZmuRSSlrvmbu6KG4KICwgqyMz5/XFeRkcGnIGBMzD357rmcuaZc858z8zU3JzznOeRCYIggIiIiMiEmEldABEREVF5YwAiIiIik8MARERERCaHAYiIiIhMDgMQERERmRwGICIiIjI5DEBERERkchiAiIiIyOQwABEREZHJYQAiMoD+/fvD19e3ROtOmTIFMpnMsAUZmatXr0Imk2HlypXl+rq7du2CTCbDrl271G26flZlVbOvry/69+9v0G0Skf4YgKhSk8lkOt2e/4EkKq19+/ZhypQpePDggdSlEFERzKUugKgs/fDDDxqPv//+eyQmJhZqr1evXqleZ+nSpVCpVCVad8KECRg3blypXp90V5rPSlf79u3D1KlT0b9/f1StWlXjufPnz8PMjH97EkmNAYgqtQ8++EDj8YEDB5CYmFio/UWPHj2CtbW1zq9TpUqVEtUHAObm5jA353+K5aU0n5UhKBQKSV+/osjJyYGNjY3UZVAlxj9DyOS1a9cODRs2xJEjR/Daa6/B2toaX3zxBQDgl19+wdtvvw0PDw8oFAr4+/tj2rRpUCqVGtt4sV9JQf+R2bNnY8mSJfD394dCoUCzZs1w+PBhjXW19QGSyWQYNmwYNm3ahIYNG0KhUKBBgwbYvn17ofp37dqFpk2bwtLSEv7+/vjvf/+rc7+i3bt3o2fPnqhRowYUCgW8vb3x2Wef4fHjx4X2z9bWFikpKQgPD4etrS2qV6+O0aNHF3ovHjx4gP79+8PBwQFVq1ZFZGSkTqeC/vnnH8hkMqxatarQc7///jtkMhl+++03AMC1a9fw6aefok6dOrCysoKTkxN69uyJq1evvvR1tPUB0rXmkydPon///qhZsyYsLS3h5uaGDz/8EHfv3lUvM2XKFHz++ecAAD8/P/Vp1oLatPUBunLlCnr27Ilq1arB2toaLVq0wJYtWzSWKejP9NNPP2H69Onw8vKCpaUl2rdvj0uXLr10v/V5zx48eIDPPvsMvr6+UCgU8PLyQr9+/ZCRkaFe5smTJ5gyZQpeeeUVWFpawt3dHd26dcPly5c16n3x9LK2vlUF36/Lly+jc+fOsLOzQ58+fQDo/h0FgHPnzuG9995D9erVYWVlhTp16mD8+PEAgJ07d0Imk2Hjxo2F1luzZg1kMhn279//0veRKg/+2UkE4O7du+jUqRN69eqFDz74AK6urgCAlStXwtbWFtHR0bC1tcWff/6JSZMmISsrC7NmzXrpdtesWYOHDx/ik08+gUwmw8yZM9GtWzdcuXLlpUci9uzZg4SEBHz66aews7PDvHnz0L17d1y/fh1OTk4AgGPHjqFjx45wd3fH1KlToVQq8eWXX6J69eo67ff69evx6NEjDBkyBE5OTjh06BDmz5+PmzdvYv369RrLKpVKhIaGIjg4GLNnz8aOHTvwzTffwN/fH0OGDAEACIKALl26YM+ePRg8eDDq1auHjRs3IjIy8qW1NG3aFDVr1sRPP/1UaPn4+Hg4OjoiNDQUAHD48GHs27cPvXr1gpeXF65evYpFixahXbt2OHPmjF5H7/SpOTExEVeuXMGAAQPg5uaGf//9F0uWLMG///6LAwcOQCaToVu3brhw4QLWrl2Lb7/9Fs7OzgBQ5Gdy+/ZttGrVCo8ePcKIESPg5OSEVatW4d1338WGDRvQtWtXjeW/+uormJmZYfTo0cjMzMTMmTPRp08fHDx4sNj91PU9y87ORkhICM6ePYsPP/wQTZo0QUZGBjZv3oybN2/C2dkZSqUS77zzDpKSktCrVy9ERUXh4cOHSExMxOnTp+Hv76/z+18gPz8foaGhaNOmDWbPnq2uR9fv6MmTJxESEoIqVapg0KBB8PX1xeXLl/Hrr79i+vTpaNeuHby9vbF69epC7+nq1avh7++Pli1b6l03VWACkQkZOnSo8OLXvm3btgIAYfHixYWWf/ToUaG2Tz75RLC2thaePHmibouMjBR8fHzUj5OTkwUAgpOTk3Dv3j11+y+//CIAEH799Vd12+TJkwvVBECwsLAQLl26pG47ceKEAECYP3++ui0sLEywtrYWUlJS1G0XL14UzM3NC21TG237FxsbK8hkMuHatWsa+wdA+PLLLzWWffXVV4WgoCD1402bNgkAhJkzZ6rb8vPzhZCQEAGAsGLFimLriYmJEapUqaLxnuXm5gpVq1YVPvzww2Lr3r9/vwBA+P7779VtO3fuFAAIO3fu1NiX5z8rfWrW9rpr164VAAh///23um3WrFkCACE5ObnQ8j4+PkJkZKT68ciRIwUAwu7du9VtDx8+FPz8/ARfX19BqVRq7Eu9evWE3Nxc9bJz584VAAinTp0q9FrP0/U9mzRpkgBASEhIKLS8SqUSBEEQli9fLgAQ5syZU+Qy2t57QXj238bz72vB92vcuHE61a3tO/raa68JdnZ2Gm3P1yMI4vdLoVAIDx48ULelp6cL5ubmwuTJkwu9DlVuPAVGBLFfxoABAwq1W1lZqe8/fPgQGRkZCAkJwaNHj3Du3LmXbjciIgKOjo7qxyEhIQDEUx4v06FDB42/pBs1agR7e3v1ukqlEjt27EB4eDg8PDzUy9WqVQudOnV66fYBzf3LyclBRkYGWrVqBUEQcOzYsULLDx48WONxSEiIxr5s3boV5ubm6iNCACCXyzF8+HCd6omIiMDTp0+RkJCgbvvjjz/w4MEDREREaK376dOnuHv3LmrVqoWqVavi6NGjOr1WSWp+/nWfPHmCjIwMtGjRAgD0ft3nX7958+Zo06aNus3W1haDBg3C1atXcebMGY3lBwwYAAsLC/VjXb9Tur5nP//8MwIDAwsdJQGgPq36888/w9nZWet7VJohHZ7/DLTVXdR39M6dO/j777/x4YcfokaNGkXW069fP+Tm5mLDhg3qtvj4eOTn57+0XyBVPgxARAA8PT01flQK/Pvvv+jatSscHBxgb2+P6tWrq/9HmZmZ+dLtvvg/44IwdP/+fb3XLVi/YN309HQ8fvwYtWrVKrSctjZtrl+/jv79+6NatWrqfj1t27YFUHj/LC0tC53Geb4eQOxn4u7uDltbW43l6tSpo1M9gYGBqFu3LuLj49Vt8fHxcHZ2xhtvvKFue/z4MSZNmgRvb28oFAo4OzujevXqePDggU6fy/P0qfnevXuIioqCq6srrKysUL16dfj5+QHQ7ftQ1Otre62CKxOvXbum0V7S75Su79nly5fRsGHDYrd1+fJl1KlTx6Cd983NzeHl5VWoXZfvaEH4e1nddevWRbNmzbB69Wp12+rVq9GiRQud/5uhyoN9gIig+VdmgQcPHqBt27awt7fHl19+CX9/f1haWuLo0aMYO3asTpdSy+Vyre2CIJTpurpQKpV48803ce/ePYwdOxZ169aFjY0NUlJS0L9//0L7V1Q9hhYREYHp06cjIyMDdnZ22Lx5M3r37q3xYzt8+HCsWLECI0eORMuWLeHg4ACZTIZevXqV6SXu7733Hvbt24fPP/8cjRs3hq2tLVQqFTp27Fjml9YXKOn3orzfs6KOBL3Yab6AQqEoNDyAvt9RXfTr1w9RUVG4efMmcnNzceDAASxYsEDv7VDFxwBEVIRdu3bh7t27SEhIwGuvvaZuT05OlrCqZ1xcXGBpaan1CiBdrgo6deoULly4gFWrVqFfv37q9sTExBLX5OPjg6SkJGRnZ2scUTl//rzO24iIiMDUqVPx888/w9XVFVlZWejVq5fGMhs2bEBkZCS++eYbdduTJ09KNPCgrjXfv38fSUlJmDp1KiZNmqRuv3jxYqFt6nMayMfHR+v7U3CK1cfHR+dtFUfX98zf3x+nT58udlv+/v44ePAgnj59WmRn/oIjUy9u/8UjWsXR9Ttas2ZNAHhp3QDQq1cvREdHY+3atXj8+DGqVKmicXqVTAdPgREVoeAv7ef/ss7Ly8N3330nVUka5HI5OnTogE2bNuHWrVvq9kuXLmHbtm06rQ9o7p8gCJg7d26Ja+rcuTPy8/OxaNEidZtSqcT8+fN13ka9evUQEBCA+Ph4xMfHw93dXSOAFtT+4hGP+fPnF3l0wRA1a3u/ACAuLq7QNgvGr9ElkHXu3BmHDh3SuAQ7JycHS5Ysga+vL+rXr6/rrhRL1/ese/fuOHHihNbLxQvW7969OzIyMrQeOSlYxsfHB3K5HH///bfG8/r896Prd7R69ep47bXXsHz5cly/fl1rPQWcnZ3RqVMn/Pjjj1i9ejU6duyovlKPTAuPABEVoVWrVnB0dERkZCRGjBgBmUyGH374wWCnoAxhypQp+OOPP9C6dWsMGTIESqUSCxYsQMOGDXH8+PFi161bty78/f0xevRopKSkwN7eHj///LNO/ZOKEhYWhtatW2PcuHG4evUq6tevj4SEBL37x0RERGDSpEmwtLTEwIEDC50aeeedd/DDDz/AwcEB9evXx/79+7Fjxw718ABlUbO9vT1ee+01zJw5E0+fPoWnpyf++OMPrUcEg4KCAADjx49Hr169UKVKFYSFhWkd2G/cuHFYu3YtOnXqhBEjRqBatWpYtWoVkpOT8fPPPxts1Ghd37PPP/8cGzZsQM+ePfHhhx8iKCgI9+7dw+bNm7F48WIEBgaiX79++P777xEdHY1Dhw4hJCQEOTk52LFjBz799FN06dIFDg4O6NmzJ+bPnw+ZTAZ/f3/89ttvSE9P17lmfb6j8+bNQ5s2bdCkSRMMGjQIfn5+uHr1KrZs2VLov4V+/fqhR48eAIBp06bp/2ZS5VDu150RSaioy+AbNGigdfm9e/cKLVq0EKysrAQPDw9hzJgxwu+///7SS6sLLvWdNWtWoW0C0LjktqjL4IcOHVpo3RcvoRYEQUhKShJeffVVwcLCQvD39xf+7//+Txg1apRgaWlZxLvwzJkzZ4QOHToItra2grOzs/Dxxx+rL7d/8TJlGxubQutrq/3u3btC3759BXt7e8HBwUHo27evcOzYMZ0ugy9w8eJFAYAAQNizZ0+h5+/fvy8MGDBAcHZ2FmxtbYXQ0FDh3Llzhd4fXS6D16fmmzdvCl27dhWqVq0qODg4CD179hRu3bpV6DMVBEGYNm2a4OnpKZiZmWlcEq/tM7x8+bLQo0cPoWrVqoKlpaXQvHlz4bffftNYpmBf1q9fr9Gu7bJybXR9zwrej2HDhgmenp6ChYWF4OXlJURGRgoZGRnqZR49eiSMHz9e8PPzE6pUqSK4ubkJPXr0EC5fvqxe5s6dO0L37t0Fa2trwdHRUfjkk0+E06dP6/z9EgTdv6OCIAinT59Wfz6WlpZCnTp1hIkTJxbaZm5uruDo6Cg4ODgIjx8/LvZ9o8pLJghG9OcsERlEeHg4/v33X639U4hMXX5+Pjw8PBAWFoZly5ZJXQ5JhH2AiCq4F6cEuHjxIrZu3Yp27dpJUxCRkdu0aRPu3Lmj0bGaTA+PABFVcO7u7ur5qa5du4ZFixYhNzcXx44dQ+3ataUuj8hoHDx4ECdPnsS0adPg7Oxc4sErqXJgJ2iiCq5jx45Yu3Yt0tLSoFAo0LJlS8yYMYPhh+gFixYtwo8//ojGjRtrTMZKpolHgIiIiMjksA8QERERmRwGICIiIjI57AOkhUqlwq1bt2BnZ1eqmY2JiIio/AiCgIcPH8LDw+Olg4gyAGlx69YteHt7S10GERERlcCNGzfg5eVV7DIMQFrY2dkBEN9Ae3t7iashIiIiXWRlZcHb21v9O14cBiAtCk572dvbMwARERFVMLp0X2EnaCIiIjI5DEBERERkchiAiIiIyOQwABEREZHJYQAiIiIik8MARERERCaHAYiIiIhMDgMQERERmRwGICIiIjI5HAmaiIiIyoVSCezeDaSmAu7uQEgIIJdLUwsDEBEREZW5hAQgKgq4efNZm5cXMHcu0K1b+dfDU2BERERUphISgB49NMMPAKSkiO0JCeVfEwMQERERlRmlUjzyIwiFnytoGzlSXK48MQARERFRmdm9u/CRn+cJAnDjhrhceWIfICIiIiNnTJ2H9ZWaatjlDIUBiIiIyIgZW+dhfbm7G3Y5Q+EpMCIiIiNljJ2H9RUSIgY2mUz78zIZ4O0tLleeGICIiIiMkLF2HtaXXC4erQIKh6CCx3Fx5X9KjwGIiIjICBlr5+GS6NYN2LAB8PTUbPfyEtulOJXHPkBERERGyFg7D5dUt25Aly7G05mbAYiIiMgIGWvn4dKQy4F27aSuQsRTYEREREbIWDsPVxaSB6CFCxfC19cXlpaWCA4OxqFDh4pc9unTp/jyyy/h7+8PS0tLBAYGYvv27aXaJhERkTEy1s7DlYWkASg+Ph7R0dGYPHkyjh49isDAQISGhiI9PV3r8hMmTMB///tfzJ8/H2fOnMHgwYPRtWtXHDt2rMTbJCIiMlbG2Hm4spAJgrYL7MpHcHAwmjVrhgULFgAAVCoVvL29MXz4cIwbN67Q8h4eHhg/fjyGDh2qbuvevTusrKzw448/lmib2mRlZcHBwQGZmZmwt7cv7W4SERGVSkUeCbo86fP7LVkn6Ly8PBw5cgQxMTHqNjMzM3To0AH79+/Xuk5ubi4sLS012qysrLBnz54Sb7Ngu7m5uerHWVlZJdonIiIyLpUlOBhT5+HKQrJTYBkZGVAqlXB1ddVod3V1RVpamtZ1QkNDMWfOHFy8eBEqlQqJiYlISEhA6v+uASzJNgEgNjYWDg4O6pu3t3cp946IiKSWkAD4+gKvvw68/774r69vxRg9mcqe5J2g9TF37lzUrl0bdevWhYWFBYYNG4YBAwbAzKx0uxETE4PMzEz17caNGwaqmIiIpFAZppCgsiVZAHJ2doZcLsft27c12m/fvg03Nzet61SvXh2bNm1CTk4Orl27hnPnzsHW1hY1a9Ys8TYBQKFQwN7eXuNGREQVU2WZQoLKlmQByMLCAkFBQUhKSlK3qVQqJCUloWXLlsWua2lpCU9PT+Tn5+Pnn39Gly5dSr1NIiKqHCrTFBJUdiQdCTo6OhqRkZFo2rQpmjdvjri4OOTk5GDAgAEAgH79+sHT0xOxsbEAgIMHDyIlJQWNGzdGSkoKpkyZApVKhTFjxui8TSIiqtwq2xQSVDYkDUARERG4c+cOJk2ahLS0NDRu3Bjbt29Xd2K+fv26Rv+eJ0+eYMKECbhy5QpsbW3RuXNn/PDDD6hatarO2yQiosqtMk4hQYYn6ThAxorjABERVVxKpXi1V0qK9n5AMpk4kGBycsW8JJ6Kps/vd4W6CoyIiMqHUgns2gWsXSv+W5E6DHMKCdIFAxAREWmoDOPncAoJehmeAtOCp8CIyFQVjJ/z4i9DwZGTihYeKstI0KQbfX6/GYC0YAAiIlNU0HemqEvI2XeGjB37ABERkd44fg6ZEgYgIiICwPFzyLQwABEREQCOn0OmhQGIiIgAiB2EvbwKXzpeQCYDvL3F5YgqOgYgIiICwPFzyLQwABERkRrHzyFTIelcYEREZHy6dQO6dOH4OVS5MQAREVEhcjnQrp3UVRCVHQYgIiID4sjDRBUDAxARkYEkJABRUZqDCXp5iR2L2XeGyLiwEzQRkQEUzKH14kjKKSlie0WaSJTIFDAAERGVklIpHvnRNrNiQdvIkeJyRGQcGICIiEqJc2gRVTwMQEREpcQ5tIgqHgYgIqJS4hxaRBUPAxARUSlxDi2iiocBiIiolDiHFlHFwwBERGQAnEOLqGLhQIhERAbCObSIKg4GICIiA+IcWkQVA0+BERERkclhACIiIiKTw1NgRGQUOIs6EZUnBiAikhxnUSei8sZTYEQkKc6iTkRSYAAiIslwFnUikgoDEBFJhrOoE5FUGICISDKcRZ2IpMIARESS4SzqRCQVBiAikgxnUSciqTAAEZFkOIs6EUmFAYiIJMVZ1IlIChwIkYgkx1nUiai8MQARkVHgLOpEldvTp0ByMnDhAnDxItCyJdCihXT1MAARERGRQSiVwLVrYsC5ePFZ2Ll4Ebh6VXNQ0y++YAAiIiKiCkKlEqeqeT7cFNwuXxaP9BTF2hqoXVu8NWxYfjVrwwBEREREGgQBSEvTfiTn0iXgyZOi11UoAH9/MeS88sqzwFO7NuDhUfSwF+WNAYiIiMgECQKQkVH4KM6FC2LIyc4uel1zc6BmTc1wUxB2vLwqxgUMDEBERESV2IMH2o/kXLwoPlcUMzPA11cz5BQEHR8fMQRVZBW8fCIiIsrO1n4k5+JF8ShPcby9tZ+u8vMTT2dVVgxARERUaT15Ih7lKO50TkXy6JHY0fjFIzkvmzDY3V376Sp/f8DKqnxqNzYMQEQVnFLJAQSp8hIEMbzcv//s9uCB7veL66xb2Tg7Fz6KU7s2UKsWYGcndXXGhwGIqAJLSACiooCbN5+1eXmJ82txCgkyFvn5QGZmyQLMgweaY8eUhEwG2Noaz9VHpVGlyrPOxy+GnapVpa6uYmEAIqqgEhKAHj3Ev5Cfl5IitnMeLTKkJ0/0CzDPP374sPSvb2EBODqKt6pVX37/+cd2dmKHXqLnyQThxf99UlZWFhwcHJCZmQl7e3upyyEqRKkUr854/sjP82Qy8UhQcjJPh9HLCQKQni5e+vzi7eZNMcTk5pb+dWxtdQ8wL4YZS8vKcQSHypY+v988AkRUAe3eXXT4AcQftBs3xOU4vxYB4ui9t25pDzmXL+vWSdjMTAwm+gaYgnWqVCnDHSTSEwMQUQX0sis+9F2OKgelUgy+RYWc4joEy2Ti2C61aolXBtWqJd5q1ACqVeOpJKp8GICIKiB3d8MuRxXH06fipJLaQk5ycvHzMMnl4tguBeHm+Zuvb+Ue84XoRZIHoIULF2LWrFlIS0tDYGAg5s+fj+bNmxe5fFxcHBYtWoTr16/D2dkZPXr0QGxsLCwtLQEAU6ZMwdSpUzXWqVOnDs6dO1em+0FUnkJCxD4+KSmFO0EDz/oAhYSUf21Uek+eiGFGW8i5dq34q6IsLDSP4Dx/RKdGDZ6GIiogaQCKj49HdHQ0Fi9ejODgYMTFxSE0NBTnz5+Hi4tLoeXXrFmDcePGYfny5WjVqhUuXLiA/v37QyaTYc6cOerlGjRogB07dqgfm1f08bqJXiCXi5e69+ghhp3nQ1BBR9G4OHaANmY5OeJpqaI6Hhd3eYq1deGQU3Dz9OTnTqQLSZPBnDlz8PHHH2PAgAEAgMWLF2PLli1Yvnw5xo0bV2j5ffv2oXXr1nj//fcBAL6+vujduzcOHjyosZy5uTnc3NzKfgeIJNStm3ipu7ZxgOLieAm8McjMLDrkvKx/lr299oBTqxbg5sYroohKS7IAlJeXhyNHjiAmJkbdZmZmhg4dOmD//v1a12nVqhV+/PFHHDp0CM2bN8eVK1ewdetW9O3bV2O5ixcvwsPDA5aWlmjZsiViY2NRo0aNImvJzc1F7nPXeGZlZZVy74jKR7duQJculWMk6IcPgUOHgNu3xVM8KpX2f0v6XHkvc/cucOdO8fvs5FT4NFXBzdmZIYeoLEkWgDIyMqBUKuHq6qrR7urqWmR/nffffx8ZGRlo06YNBEFAfn4+Bg8ejC+++EK9THBwMFauXIk6deogNTUVU6dORUhICE6fPg27IsYCj42NLdRviKiikMsr5qXuN24Ae/c+u504IQaHysbVVftRHH9/8coqIpJGheocs2vXLsyYMQPfffcdgoODcenSJURFRWHatGmYOHEiAKBTp07q5Rs1aoTg4GD4+Pjgp59+wsCBA7VuNyYmBtHR0erHWVlZ8Pb2LtudITIhSiVw8qRm4Llxo/ByBZdhy+Xi5dbP/6utTZfnDLWMvuvb24shh3MwERknyQKQs7Mz5HI5bt++rdF++/btIvvvTJw4EX379sVHH30EAAgICEBOTg4GDRqE8ePHw0zLABVVq1bFK6+8gkuXLhVZi0KhgILXfxIZzMOHwIEDz8LOgQOFB9qTy4HAQKB1a6BNG6BVK7H/EhFReZAsAFlYWCAoKAhJSUkIDw8HAKhUKiQlJWHYsGFa13n06FGhkCP/X2eHomb0yM7OxuXLlwv1EyIiw9HldJadHdCypRh4WrcGgoPFqRGIiKQg6Smw6OhoREZGomnTpmjevDni4uKQk5OjviqsX79+8PT0RGxsLAAgLCwMc+bMwauvvqo+BTZx4kSEhYWpg9Do0aMRFhYGHx8f3Lp1C5MnT4ZcLkfv3r0l20+iyiQ/Hzh1SrfTWQVhp3VroGHDitk5m4gqJ0kDUEREBO7cuYNJkyYhLS0NjRs3xvbt29Udo69fv65xxGfChAmQyWSYMGECUlJSUL16dYSFhWH69OnqZW7evInevXvj7t27qF69Otq0aYMDBw6gevXq5b5/RJWBrqezGjfWDDyenpKUS0SkE84GrwVngydTdv265tGdkycLn86yt9c8ndW8OU9nEZH0OBs8EekkP7/w1VnaZpn39dU8utOgAU9nEVHFxgBEZEKysjRPZx08qP101quvagYeDw9p6iUiKisMQESVlCAUPp116lTh01kODoVPZ9nYSFMzEVF5YQAiqiTy88XLz58PPCkphZfz9RXH3SkIPPXr83QWEZkeBiCiCiw5Gfj1V2DLFjHw5ORoPs/TWURE2jEAEVUgSqXYb+fXX8Xbv/9qPs/TWUREumEAIjJyDx8Cf/whBp6tWzVnGJfLxaATFgaEhopXZ2mZEYaIiF7AAERkhK5de3aUZ9cuIC/v2XMODkDHjmLo6dQJqFZNsjKJiCosBiAiI6BSAYcOPQs9p05pPl+rlhh4wsLEDsxVqkhTJxFRZcEARCSR7GwgMfFZJ+b09GfPmZk9O7UVFgbUqQPIZNLVSkRU2TAAEZWjGzeeHeXZuRPIzX32nL295qktJyfp6iQiquwYgIjKkEoF/PPPs9Bz4oTm8zVrPjvKExICWFhIUycRkalhACIysJwcYMeOZ6e20tKePWdmJl6mXhB66tXjqS0iIikwAJHJUiqB3buB1FTA3V08AlPSEZFv3gR++00MPX/+CTx58uw5OzvxEvWwMKBzZ8DZ2TD1ExFRyTEAkUlKSACiojRnPvfyAubOBbp1e/n6KhVw9OizU1vHjmk+7+v77ChP27Y8tUVEZGwYgMjkJCQAPXqIk4U+LyVFbN+wQXsIevQISEoSA89vv4lHjgrIZECLFs9CT4MGPLVFRGTMZILw4s8AZWVlwcHBAZmZmbC3t5e6HDIgpVI8OvP8kZ/nyWTikaDkZPF02K1bz05tJSUBjx8/W9bGRvPUlotLuewCEREVQZ/fbx4BIpOye3fR4QcQjwrduAF89JE4GOGRI5rP16jx7ChPu3aAQlGm5RIRURlhACKT8vxpq+KsXCn+K5OJE4oWhJ6AAJ7aIiKqDBiAyKS4u+u2XJs2wIABwNtvA66uZVsTERGVPwYgMilt2oiXoWdkFL2Ml5c4AWlJL4knIiLjZyZ1AUTlQakUr+5q3rzo8COTibe5cxl+iIgqOwYgqtSePhX78zRoAPTsKY7XY20NvPNO4dNhXl5FXwJPRESVC0+BUaX06BGwbBkwezZw/brYVrUqMGIEMHy4eBrMkCNBExFRxcIARJVKZibw3XfAt98Cd+6Iba6uwKhRwCefiDOuF5DLxUvZiYjI9DAAUaWQni723VmwAMjKEtt8fYExY8SruSwtJS2PiIiMDAMQVWg3boinuZYufTZKc716QEwM0KsXUKWKtPUREZFxYgCiCunCBeDrr4EffhA7OgNA06bA+PHAu+8CZuzeT0RExWAAogrl2DEgNla8WqtgFrvXXwe++AJo356jNBMRkW4YgKhC2LMHmDED2LbtWVtYmHiqq2VL6eoiIqKKiQGIjJYgAL//Lgaf3bvFNjMzICICGDcOaNRI2vqIiKjiYgAio6NUAhs3isHn2DGxzcIC6N8f+PxzoFYtScsjIqJKgAGIjMbTp8Dq1cBXXwHnz4tt1tbA4MFAdDTg6SltfUREVHkwAJHkHj8WR22eNavoUZuJiIgMiQGIJFMwanNcnDiQIQC4uYlHewYPBuzsJC2PiIgqMQYgKndFjdo8dqzYz4ejNhMRUVljAKJyo23U5vr1xUvZIyI4ajMREZUfBiAqc9pGbW7WTBy8kKM2ExGRFBiAqMwcPy6O2rx+/bNRm994Qzziw1GbiYhISgxAZHB79ojBZ+vWZ23vvisGnxYtpKuLiIioAAMQGURRozb36iWO2hwQIG19REREz2MAolIpbtTmMWMAf39JyyMiItKKAYhKhKM2ExFRRcYARHpLThY7MScni48dHcURm0eMAJycpK2NiIhIFwxApLdJk8Tw4+oKjB4NfPIJR20mIqKKhQGI9JKcDKxdK97fsgUICpK2HiIiopLgEHSkl9mzxY7Pb77J8ENERBUXAxDp7PZtYPly8X5MjLS1EBERlQYDEOksLg548gQIDgbatZO6GiIiopJjACKdZGYC330n3o+J4TQWRERUsTEAkU6++w7IyhJnbw8Lk7oaIiKi0pE8AC1cuBC+vr6wtLREcHAwDh06VOzycXFxqFOnDqysrODt7Y3PPvsMT548KdU2qXiPH4unvwBxWgvO3k5ERBWdpD9l8fHxiI6OxuTJk3H06FEEBgYiNDQU6enpWpdfs2YNxo0bh8mTJ+Ps2bNYtmwZ4uPj8cUXX5R4m/Ryy5cD6emAj484txcREVFFJxMEQZDqxYODg9GsWTMsWLAAAKBSqeDt7Y3hw4dj3LhxhZYfNmwYzp49i6SkJHXbqFGjcPDgQezZs6dE29QmKysLDg4OyMzMhL29fWl3s0J7+hSoXRu4dg1YsAAYOlTqioiIiLTT5/dbsiNAeXl5OHLkCDp06PCsGDMzdOjQAfv379e6TqtWrXDkyBH1Ka0rV65g69at6Ny5c4m3ScVbt04MPy4uwIcfSl0NERGRYUg2EnRGRgaUSiVcXV012l1dXXHu3Dmt67z//vvIyMhAmzZtIAgC8vPzMXjwYPUpsJJsEwByc3ORm5urfpyVlVXS3apUVCpxslMAGDkSsLKStBwiIiKDqVDdWXft2oUZM2bgu+++w9GjR5GQkIAtW7Zg2rRppdpubGwsHBwc1Ddvb28DVVyx/forcOYMYG8PfPqp1NUQEREZjt4ByNfXF19++SWuX79eqhd2dnaGXC7H7du3Ndpv374NNzc3retMnDgRffv2xUcffYSAgAB07doVM2bMQGxsLFQqVYm2CQAxMTHIzMxU327cuFGqfasMBAGIjRXvf/op4OAgbT1ERESGpHcAGjlyJBISElCzZk28+eabWLduncbpI11ZWFggKChIo0OzSqVCUlISWrZsqXWdR48eweyFa7DlcjkAQBCEEm0TABQKBezt7TVupm7XLuDgQcDSUjz9RUREVJmUKAAdP34chw4dQr169TB8+HC4u7tj2LBhOHr0qF7bio6OxtKlS7Fq1SqcPXsWQ4YMQU5ODgYMGAAA6NevH2Kem3QqLCwMixYtwrp165CcnIzExERMnDgRYWFh6iD0sm2SbgqO/gwcCLzQpYqIiKjiE0opLy9PiIuLExQKhWBmZiYEBgYKy5YtE1QqlU7rz58/X6hRo4ZgYWEhNG/eXDhw4ID6ubZt2wqRkZHqx0+fPhWmTJki+Pv7C5aWloK3t7fw6aefCvfv39d5m7rIzMwUAAiZmZl6rVdZ/POPIACCIJcLQnKy1NUQERHpRp/f7xKPA/T06VNs3LgRK1asQGJiIlq0aIGBAwfi5s2bWLhwId544w2sWbPGsGmtnJj6OEA9egA//wz07Qt8/73U1RAREelGn99vvS+DP3r0KFasWIG1a9fCzMwM/fr1w7fffou6deuql+natSuaNWumf+UkuXPngIQE8f7YsdLWQkREVFb0DkDNmjXDm2++iUWLFiE8PBxVqlQptIyfnx96cc6ECmnmTPEKsC5dgAYNpK6GiIiobOh9CuzatWvw8fEpq3qMgqmeArtxA6hZE8jPBw4cAIKDpa6IiIhId2U6FUZ6ejoOHjxYqP3gwYP4559/9N0cGZFvvhHDz+uvM/wQEVHlpncAGjp0qNaBAlNSUjCUM2VWWBkZwNKl4v3nRh4gIiKqlPQOQGfOnEGTJk0Ktb/66qs4c+aMQYqi8jdvHvDoERAUBDw3lywREVGlpHcAUigUhaaaAIDU1FSYm0s2tyqVwsOHwPz54v2YGEAmk7YeIiKisqZ3AHrrrbfUc2cVePDgAb744gu8+eabBi2Oysd//ws8eADUqQN07Sp1NURERGVP70M2s2fPxmuvvQYfHx+8+uqrAIDjx4/D1dUVP/zwg8ELpLKVmwvMmSPeHzsWMNM7EhMREVU8egcgT09PnDx5EqtXr8aJEydgZWWFAQMGoHfv3lrHBCLjtmoVkJoKeHkBffpIXQ0REVH5KFGnHRsbGwwaNMjQtVA5y88XBz4EgNGjAQsLaeshIiIqLyXutXzmzBlcv34deXl5Gu3vvvtuqYui8rFhA3D5MuDkBHz0kdTVEBERlR+9A9CVK1fQtWtXnDp1CjKZDAUDScv+d+mQUqk0bIVUJgQB+Oor8X5UFGBjI209RERE5UnvLq9RUVHw8/NDeno6rK2t8e+//+Lvv/9G06ZNsWvXrjIokcrCtm3AiROArS0wbJjU1RAREZUvvY8A7d+/H3/++SecnZ1hZmYGMzMztGnTBrGxsRgxYgSOHTtWFnWSgcXGiv8OHgw4OkpbCxERUXnT+wiQUqmEnZ0dAMDZ2Rm3bt0CAPj4+OD8+fOGrY7KxO7dwJ49Yqfnzz6TuhoiIqLyp/cRoIYNG+LEiRPw8/NDcHAwZs6cCQsLCyxZsgQ1a9YsixrJwAqO/vTvD3h46L++UimGqNRUwN0dCAkB5HKDlkhERFSm9A5AEyZMQE5ODgDgyy+/xDvvvIOQkBA4OTkhPj7e4AWSYR0/Lvb/MTMDxozRf/2EBLHT9M2bz9q8vIC5c4Fu3QxWJhERUZmSCQWXcZXCvXv34OjoqL4SrKLLysqCg4MDMjMzYW9vL3U5BtWrFxAfL/67dq1+6yYkAD16iFeQPa/gY9+wgSGIiIiko8/vt159gJ4+fQpzc3OcPn1ao71atWqVJvxUZpcuAevXi/fHjdNvXaVSPPKjLS4XtI0cKS5HRERk7PQKQFWqVEGNGjU41k8FNXMmoFIBnTsDgYH6rbt7t+ZprxcJAnDjhrgcERGRsdP7KrDx48fjiy++wL1798qiHiojt26J834BQEyM/uunphp2OSIiIinp3Ql6wYIFuHTpEjw8PODj4wObF4YQPnr0qMGKI8OZMwfIywPatBFv+nJ3N+xyREREUtI7AIWHh5dBGVSW7t0DFi8W75fk6A8gXuru5QWkpGjvBySTic+HhJS8TiIiovKidwCaPHlyWdRBZWjBAiAnR+z306lTybYhl4uXuvfoIYad50NQQf/3uDiOB0RERBWD3n2AqGLJyQHmzRPvjxv3LKyURLdu4qXunp6a7V5evASeiIgqFr2PAJmZmRV7yTuvEDMuS5cCd+8C/v7i0ZvS6tYN6NKFI0ETEVHFpncA2rhxo8bjp0+f4tixY1i1ahWmTp1qsMKo9PLygG++Ee+PGQOY6/1payeXA+3aGWZbREREUjDISNAAsGbNGsTHx+OXX34xxOYkVVlGgl6+HBg4UDxKk5wMKBRSV0RERFR2ymwk6OK0aNECSUlJhtoclZJSCXz9tXg/Oprhh4iI6HkGCUCPHz/GvHnz4Pli71iSzMaNwIULgKMj8MknUldDRERkXPTuFfLipKeCIODhw4ewtrbGjz/+aNDiqGQEAYiNFe8PGwbY2UlbDxERkbHROwB9++23GgHIzMwM1atXR3BwMBwdHQ1aHJVMYiJw9ChgbQ2MGCF1NURERMZH7wDUv3//MiiDDKng6M/HHwPOztLWQkREZIz07gO0YsUKrF+/vlD7+vXrsapgtk2SzIEDwK5dQJUqwKhRUldDRERknPQOQLGxsXDWcljBxcUFM2bMMEhRVHIFR38++ADw9pa2FiIiImOldwC6fv06/Pz8CrX7+Pjg+vXrBimKSub0aWDzZnG6i7Fjpa6GiIjIeOkdgFxcXHDy5MlC7SdOnICTk5NBiqKSKRj3p3t3oE4daWshIiIyZnoHoN69e2PEiBHYuXMnlEollEol/vzzT0RFRaFXr15lUSPpIDkZWLtWvB8TI20tRERExk7vq8CmTZuGq1evon379jD/3+RSKpUK/fr1Yx8gCc2eLY7+/NZbQJMmUldDRERk3Eo8F9jFixdx/PhxWFlZISAgAD4+PoauTTIVbS6w27cBX1/gyRNg505OVEpERKZJn9/vEs8PXrt2bdSuXbukq5MBxcWJ4adFC6BtW6mrISIiMn569wHq3r07vi7obfucmTNnomfPngYpinSXmQl89514PyZGvAKMiIiIiqd3APr777/RuXPnQu2dOnXC33//bZCiSHfffQdkZQENGgDvvCN1NURERBWD3gEoOzsbFhYWhdqrVKmCrKwsgxRFunn8WDz9BQDjxgFmen+aREREpknvn8yAgADEx8cXal+3bh3q169vkKJIN8uXA+npYgdojkBARESkO707QU+cOBHdunXD5cuX8cYbbwAAkpKSsGbNGmzYsMHgBZJ2T58Cs2aJ9z//HDAvcXd2IiIi06P3z2ZYWBg2bdqEGTNmYMOGDbCyskJgYCD+/PNPVKtWrSxqJC3WrQOuXQNcXIABA6SuhoiIqGIp0XGDt99+G2+//TYA8Zr7tWvXYvTo0Thy5AiUSqVBC6TCVCrgq6/E+599BlhZSVsPERFRRVPibrN///03IiMj4eHhgW+++QZvvPEGDhw4YMjaqAi//gqcOQPY2wNDhkhdDRERUcWj1xGgtLQ0rFy5EsuWLUNWVhbee+895ObmYtOmTewAXU4EAYiNFe8PHQo4OEhbDxERUUWk8xGgsLAw1KlTBydPnkRcXBxu3bqF+fPnl2VtpMWuXcDBg4ClJTBypNTVEBERVUw6B6Bt27Zh4MCBmDp1Kt5++23I5XKDFbFw4UL4+vrC0tISwcHBOHToUJHLtmvXDjKZrNCtoE8SAPTv37/Q8x07djRYvVIqmG924ECxAzQRERHpT+cAtGfPHjx8+BBBQUEIDg7GggULkJGRUeoC4uPjER0djcmTJ+Po0aMIDAxEaGgo0tPTtS6fkJCA1NRU9e306dOQy+WFpuHo2LGjxnJr164tda1S++cfYMcOQC4HRo+WuhoiIqKKS+cA1KJFCyxduhSpqan45JNPsG7dOnh4eEClUiExMREPHz4sUQFz5szBxx9/jAEDBqB+/fpYvHgxrK2tsXz5cq3LV6tWDW5ubupbYmIirK2tCwUghUKhsZyjo2OJ6jMmBX1/3n9fHPyQiIiISkbvq8BsbGzw4YcfYs+ePTh16hRGjRqFr776Ci4uLnj33Xf12lZeXh6OHDmCDh06PCvIzAwdOnTA/v37ddrGsmXL0KtXL9jY2Gi079q1Cy4uLqhTpw6GDBmCu3fv6lWbsTl3Dti4Ubw/dqy0tRAREVV0pZo9qk6dOpg5cyZu3rxZolNMGRkZUCqVcHV11Wh3dXVFWlraS9c/dOgQTp8+jY8++kijvWPHjvj++++RlJSEr7/+Gn/99Rc6depU5BhFubm5yMrK0rgZm6+/Fq8A69JFnPiUiIiISs4gEyjI5XKEh4cjPDzcEJvT2bJlyxAQEIDmzZtrtPd6bmKsgIAANGrUCP7+/ti1axfat29faDuxsbGYOnVqmddbUtevAz/+KN6PiZG2FiIiospA0vnDnZ2dIZfLcfv2bY3227dvw83Nrdh1c3JysG7dOgwcOPClr1OzZk04Ozvj0qVLWp+PiYlBZmam+nbjxg3dd6IcfPMNkJ8PvP46EBwsdTVEREQVn6QByMLCAkFBQUhKSlK3qVQqJCUloWXLlsWuu379euTm5uKDDz546evcvHkTd+/ehbu7u9bnFQoF7O3tNW7G4s4dYOlS8T6P/hARERmGpAEIAKKjo7F06VKsWrUKZ8+exZAhQ5CTk4MB/5vhs1+/fojR8su/bNkyhIeHw8nJSaM9Ozsbn3/+OQ4cOICrV68iKSkJXbp0Qa1atRAaGlou+2RI8+YBjx8DQUHAc33FiYiIqBQM0geoNCIiInDnzh1MmjQJaWlpaNy4MbZv367uGH39+nWYmWnmtPPnz2PPnj34448/Cm1PLpfj5MmTWLVqFR48eAAPDw+89dZbmDZtGhQKRbnsk6FkZQELFoj3Y2IAmUzaeoiIiCoLmSAIgtRFGJusrCw4ODggMzNT0tNhs2YBY8YAdeqIk5+aSX68joiIyHjp8/vNn1Qj9eQJMGeOeH/sWIYfIiIiQ+LPqpFatQpISwO8vIA+faSuhoiIqHJhADJC+fnAzJni/dGjAQsLaeshIiKqbBiAjND69cCVK4CTE/DCINdERERkAAxARkYQgK++Eu9HRQEvTHFGREREBsAAZGS2bgVOngRsbYFhw6SuhoiIqHJiADIysbHiv4MHA46O0tZCRERUWTEAGZHdu4G9e8VOz599JnU1RERElRcDkBEpOPrTvz/g4SFpKURERJUaA5CROH4c2LZNHPBwzBipqyEiIqrcGICMRMGVX++9B/j7S1sLERFRZccAZAQuXRLH/gGAceOkrYWIiMgUMAAZgZkzAZUK6NwZCAyUuhoiIqLKjwFIYrduifN+AUBMjLS1EBERmQoGIInNmQPk5QFt2og3IiIiKnsMQBK6dw9YvFi8z6M/RERE5YcBSEILFgA5OWK/n06dpK6GiIjIdDAASSQnB5g3T7wfEwPIZNLWQ0REZEoYgCSydClw9y5QqxbQo4fU1RAREZkWBiAJ5OUB33wj3h8zBpDLpa2HiIjI1DAASeDHH4GbN8X5vvr1k7oaIiIi08MAVM6USuDrr8X70dGAQiFtPURERKaIAaicbdwIXLgAODoCgwZJXQ0REZFpYgAqR4IAxMaK94cPB+zspK2HiIjIVDEAlaPERODoUcDaGhgxQupqiIiITBcDUDn65Rfx30GDACcnaWshIiIyZeZSF2BKFiwAunYF6tWTuhIiIiLTxgBUjmQyoEMHqasgIiIingIjIiIik8MARERERCaHAYiIiIhMDgMQERERmRwGICIiIjI5DEBERERkchiAiIiIyOQwABEREZHJYQAiIiIik8MARERERCaHAYiIiIhMDgMQERERmRwGICIiIjI5DEBERERkchiAiIiIyOQwABEREZHJYQAiIiIik8MARERERCaHAYiIiIhMDgMQERERmRwGICIiIjI5DEBERERkchiAiIiIyOQwABEREZHJMYoAtHDhQvj6+sLS0hLBwcE4dOhQkcu2a9cOMpms0O3tt99WLyMIAiZNmgR3d3dYWVmhQ4cOuHjxYnnsChEREVUAkgeg+Ph4REdHY/LkyTh69CgCAwMRGhqK9PR0rcsnJCQgNTVVfTt9+jTkcjl69uypXmbmzJmYN28eFi9ejIMHD8LGxgahoaF48uRJee0WERERGTGZIAiClAUEBwejWbNmWLBgAQBApVLB29sbw4cPx7hx4166flxcHCZNmoTU1FTY2NhAEAR4eHhg1KhRGD16NAAgMzMTrq6uWLlyJXr16vXSbWZlZcHBwQGZmZmwt7cv3Q4SERFRudDn91vSI0B5eXk4cuQIOnTooG4zMzNDhw4dsH//fp22sWzZMvTq1Qs2NjYAgOTkZKSlpWls08HBAcHBwUVuMzc3F1lZWRo3IiIiqrwkDUAZGRlQKpVwdXXVaHd1dUVaWtpL1z906BBOnz6Njz76SN1WsJ4+24yNjYWDg4P65u3tre+uEBERUQUieR+g0li2bBkCAgLQvHnzUm0nJiYGmZmZ6tuNGzcMVCEREREZI0kDkLOzM+RyOW7fvq3Rfvv2bbi5uRW7bk5ODtatW4eBAwdqtBesp882FQoF7O3tNW5ERERUeUkagCwsLBAUFISkpCR1m0qlQlJSElq2bFnsuuvXr0dubi4++OADjXY/Pz+4ublpbDMrKwsHDx586TaJiIjINJhLXUB0dDQiIyPRtGlTNG/eHHFxccjJycGAAQMAAP369YOnpydiY2M11lu2bBnCw8Ph5OSk0S6TyTBy5Ej85z//Qe3ateHn54eJEyfCw8MD4eHh5bVbREREZMQkD0ARERG4c+cOJk2ahLS0NDRu3Bjbt29Xd2K+fv06zMw0D1SdP38ee/bswR9//KF1m2PGjEFOTg4GDRqEBw8eoE2bNti+fTssLS3LfH+IiIjI+Ek+DpAx4jhAREREFU+FGQeIiIiISAoMQERERGRyGICIiIjI5DAAERERkclhACIiIiKTwwBEREREJocBiIiIiEwOAxARERGZHAYgIiIiMjkMQERERGRyGICIiIjI5DAAERERkclhACIiIiKTwwBEREREJocBiIiIiEwOAxARERGZHAYgIiIiMjkMQERERGRyGICIiIjI5DAAERERkclhACIiIiKTwwBEREREJocBiIiIiEwOAxARERGZHAYgIiIiMjkMQERERGRyGICIiIjI5DAAERERkclhACIiIiKTYy51AUREZFqUSiWePn0qdRlUAVWpUgVyudwg22IAIiKiciEIAtLS0vDgwQOpS6EKrGrVqnBzc4NMJivVdhiAiIioXBSEHxcXF1hbW5f6B4xMiyAIePToEdLT0wEA7u7updoeAxAREZU5pVKpDj9OTk5Sl0MVlJWVFQAgPT0dLi4upTodxk7QRERU5gr6/FhbW0tcCVV0Bd+h0vYjYwAiIqJyw9NeVFqG+g4xABEREZUjX19fxMXF6bz8rl27IJPJ2HncwNgHiIiIKgylEti9G0hNBdzdgZAQwEBXRRfysiMNkydPxpQpU/Te7uHDh2FjY6Pz8q1atUJqaiocHBz0fi0qGgMQERFVCAkJQFQUcPPmszYvL2DuXKBbN8O/Xmpqqvp+fHw8Jk2ahPPnz6vbbG1t1fcFQYBSqYS5+ct/VqtXr65XHRYWFnBzc9NrHXo5ngIjIiKjl5AA9OihGX4AICVFbE9IMPxrurm5qW8ODg6QyWTqx+fOnYOdnR22bduGoKAgKBQK7NmzB5cvX0aXLl3g6uoKW1tbNGvWDDt27NDY7ounwGQyGf7v//4PXbt2hbW1NWrXro3Nmzern3/xFNjKlStRtWpV/P7776hXrx5sbW3RsWNHjcCWn5+PESNGoGrVqnBycsLYsWMRGRmJ8PDwIvf37t276N27Nzw9PWFtbY2AgACsXbtWYxmVSoWZM2eiVq1aUCgUqFGjBqZPn65+/ubNm+jduzeqVasGGxsbNG3aFAcPHizBu1/2GICIiMioKZXikR9BKPxcQdvIkeJy5W3cuHH46quvcPbsWTRq1AjZ2dno3LkzkpKScOzYMXTs2BFhYWG4fv16sduZOnUq3nvvPZw8eRKdO3dGnz59cO/evSKXf/ToEWbPno0ffvgBf//9N65fv47Ro0ern//666+xevVqrFixAnv37kVWVhY2bdpUbA1PnjxBUFAQtmzZgtOnT2PQoEHo27cvDh06pF4mJiYGX331FSZOnIgzZ85gzZo1cHV1BQBkZ2ejbdu2SElJwebNm3HixAmMGTMGKpVKh3dSAgIVkpmZKQAQMjMzpS6FiKhSePz4sXDmzBnh8ePHeq+7c6cgiFGn+NvOnQYvW23FihWCg4PDczXtFAAImzZteum6DRo0EObPn69+7OPjI3z77bfqxwCECRMmqB9nZ2cLAIRt27ZpvNb9+/fVtQAQLl26pF5n4cKFgqurq/qxq6urMGvWLPXj/Px8oUaNGkKXLl103WVBEATh7bffFkaNGiUIgiBkZWUJCoVCWLp0qdZl//vf/wp2dnbC3bt39XoNfRX3XdLn95t9gIiIyKg9d2bHIMsZUtOmTTUeZ2dnY8qUKdiyZQtSU1ORn5+Px48fv/QIUKNGjdT3bWxsYG9vrx7xWBtra2v4+/urH7u7u6uXz8zMxO3bt9G8eXP183K5HEFBQcUejVEqlZgxYwZ++uknpKSkIC8vD7m5uepxd86ePYvc3Fy0b99e6/rHjx/Hq6++imrVqhW7r8aCAYiIiIyarjMelHJmhBJ58Wqu0aNHIzExEbNnz0atWrVgZWWFHj16IC8vr9jtVKlSReOxTCYrNqxoW17Qdo5QD7NmzcLcuXMRFxeHgIAA2NjYYOTIkeraC0ZhLsrLnjc27ANERERGLSREvNqrqKvSZTLA21tcTmp79+5F//790bVrVwQEBMDNzQ1Xr14t1xocHBzg6uqKw4cPq9uUSiWOHj1a7Hp79+5Fly5d8MEHHyAwMBA1a9bEhQsX1M/Xrl0bVlZWSEpK0rp+o0aNcPz48WL7LhkTBiAiIjJqcrl4qTtQOAQVPI6LK7vxgPRRu3ZtJCQk4Pjx4zhx4gTef/99SToBDx8+HLGxsfjll19w/vx5REVF4f79+8WObVS7dm0kJiZi3759OHv2LD755BPcvn1b/bylpSXGjh2LMWPG4Pvvv8fly5dx4MABLFu2DADQu3dvuLm5ITw8HHv37sWVK1fw888/Y//+/WW+vyXBAEREREavWzdgwwbA01Oz3ctLbC+LcYBKYs6cOXB0dESrVq0QFhaG0NBQNGnSpNzrGDt2LHr37o1+/fqhZcuWsLW1RWhoKCwtLYtcZ8KECWjSpAlCQ0PRrl07dZh53sSJEzFq1ChMmjQJ9erVQ0REhLrvkYWFBf744w+4uLigc+fOCAgIwFdffVWqCUvLkkwo7UnDSigrKwsODg7IzMyEvb291OUQEVV4T548QXJyMvz8/Ir9EX6Z8hwJujJRqVSoV68e3nvvPUybNk3qckqluO+SPr/f7ARNREQVhlwOtGsndRXG79q1a/jjjz/Qtm1b5ObmYsGCBUhOTsb7778vdWlGg6fAiIiIKhkzMzOsXLkSzZo1Q+vWrXHq1Cns2LED9erVk7o0o8EjQERERJWMt7c39u7dK3UZRo1HgIiIiMjkMAARERGRyZE8AC1cuBC+vr6wtLREcHCwxqRr2jx48ABDhw6Fu7s7FAoFXnnlFWzdulX9/JQpUyCTyTRudevWLevdICIiogpE0j5A8fHxiI6OxuLFixEcHIy4uDiEhobi/PnzcHFxKbR8Xl4e3nzzTbi4uGDDhg3w9PTEtWvXULVqVY3lGjRogB07dqgfm5uzqxMRERE9I2kymDNnDj7++GMMGDAAALB48WJs2bIFy5cvx7hx4wotv3z5cty7dw/79u1Tz4Pi6+tbaDlzc3O4ubmVae1ERERUcUl2CiwvLw9HjhxBhw4dnhVjZoYOHToUOWz25s2b0bJlSwwdOhSurq5o2LAhZsyYAaVSqbHcxYsX4eHhgZo1a6JPnz4vnYU3NzcXWVlZGjciIiKqvCQLQBkZGVAqlXB1ddVod3V1RVpamtZ1rly5gg0bNkCpVGLr1q2YOHEivvnmG/znP/9RLxMcHIyVK1di+/btWLRoEZKTkxESEoKHDx8WWUtsbCwcHBzUN29vb8PsJBERmbx27dph5MiR6se+vr6Ii4srdh2ZTIZNmzaV+rUNtZ3KSPJO0PpQqVRwcXHBkiVLEBQUhIiICIwfPx6LFy9WL9OpUyf07NkTjRo1QmhoKLZu3YoHDx7gp59+KnK7MTExyMzMVN9u3LhRHrtDRERGLCwsDB07dtT63O7duyGTyXDy5Em9t3v48GEMGjSotOVpmDJlCho3blyoPTU1FZ06dTLoa1UWkvUBcnZ2hlwu15hpFgBu375dZP8dd3d3VKlSRWNitXr16iEtLQ15eXmwsLAotE7VqlXxyiuv4NKlS0XWolAooFAoSrgnRERUGQ0cOBDdu3fHzZs34eXlpfHcihUr0LRpUzRq1Ejv7VavXt1QJb4U+8MWTbIjQBYWFggKCkJSUpK6TaVSISkpCS1bttS6TuvWrXHp0iWoVCp124ULF+Du7q41/ABAdnY2Ll++DHd3d8PuABERVWrvvPMOqlevjpUrV2q0Z2dnY/369Rg4cCDu3r2L3r17w9PTE9bW1ggICMDatWuL3e6Lp8AuXryI1157DZaWlqhfvz4SExMLrTN27Fi88sorsLa2Rs2aNTFx4kQ8ffoUALBy5UpMnToVJ06cUA//UlDzi6fATp06hTfeeANWVlZwcnLCoEGDkJ2drX6+f//+CA8Px+zZs+Hu7g4nJycMHTpU/VraXL58GV26dIGrqytsbW3RrFkzjSuxAbGv7dixY+Ht7Q2FQoFatWph2bJl6uf//fdfvPPOO7C3t4ednR1CQkJw+fLlYt/H0pL0KrDo6GhERkaiadOmaN68OeLi4pCTk6O+Kqxfv37w9PREbGwsAGDIkCFYsGABoqKiMHz4cFy8eBEzZszAiBEj1NscPXo0wsLC4OPjg1u3bmHy5MmQy+Xo3bu3JPtIRETaCQLw6FH5v661NSCTvXw5c3Nz9OvXDytXrsT48eMh+99K69evh1KpRO/evZGdnY2goCCMHTsW9vb22LJlC/r27Qt/f380b978pa+hUqnQrVs3uLq64uDBg8jMzNToL1TAzs4OK1euhIeHB06dOoWPP/4YdnZ2GDNmDCIiInD69Gls375dHTwcHBwKbSMnJwehoaFo2bIlDh8+jPT0dHz00UcYNmyYRsjbuXMn3N3dsXPnTly6dAkRERFo3LgxPv74Y637kJ2djc6dO2P69OlQKBT4/vvvERYWhvPnz6NGjRoAxN/z/fv3Y968eQgMDERycjIyMjIAACkpKXjttdfQrl07/Pnnn7C3t8fevXuRn5//0vevVASJzZ8/X6hRo4ZgYWEhNG/eXDhw4ID6ubZt2wqRkZEay+/bt08IDg4WFAqFULNmTWH69OlCfn6++vmIiAjB3d1dsLCwEDw9PYWIiAjh0qVLetWUmZkpABAyMzNLtW8vys8XhJ07BWHNGvHf58omIqrUHj9+LJw5c0Z4/Pixui07WxDEGFS+t+xs3es+e/asAEDYuXOnui0kJET44IMPilzn7bffFkaNGqV+3LZtWyEqKkr92MfHR/j2228FQRCE33//XTA3NxdSUlLUz2/btk0AIGzcuLHI15g1a5YQFBSkfjx58mQhMDCw0HLPb2fJkiWCo6OjkP3cG7BlyxbBzMxMSEtLEwRBECIjIwUfHx+N39WePXsKERERRdaiTYMGDYT58+cLgiAI58+fFwAIiYmJWpeNiYkR/Pz8hLy8PJ22re27VECf32/JRwgcNmwYhg0bpvW5Xbt2FWpr2bIlDhw4UOT21q1bZ6jSDCohAYiKAm7efNbm5QXMnQt06yZdXUREVLS6deuiVatWWL58Odq1a4dLly5h9+7d+PLLLwEASqUSM2bMwE8//YSUlBTk5eUhNzcX1tbWOm3/7Nmz8Pb2hoeHh7pNWzeQ+Ph4zJs3D5cvX0Z2djby8/Nhb2+v176cPXsWgYGBsLGxUbe1bt0aKpUK58+fV1+V3aBBA42+tu7u7jh16lSR283OzsaUKVOwZcsWpKamIj8/H48fP1YPQXP8+HHI5XK0bdtW6/rHjx9HSEiIeny/8iJ5ADIFCQlAjx7i3x7PS0kR2zdsYAgiItNjbQ081/2kXF9XHwMHDsTw4cOxcOFCrFixAv7+/uof81mzZmHu3LmIi4tDQEAAbGxsMHLkSOTl5Rms3v3796NPnz6YOnUqQkND4eDggHXr1uGbb74x2Gs878UgIpPJNPrevmj06NFITEzE7NmzUatWLVhZWaFHjx7q98DKyqrY13vZ82WFAaiMKZXikZ8Xww8gtslkwMiRQJcuwHOBm4io0pPJgOcORhit9957D1FRUVizZg2+//57DBkyRN0faO/evejSpQs++OADAGKfngsXLqB+/fo6bbtevXq4ceMGUlNT1RfrvHiWY9++ffDx8cH48ePVbdeuXdNYxsLCotCgwNpea+XKlcjJyVEfBdq7dy/MzMxQp04dnerVZu/evejfvz+6du0KQDwidPXqVfXzAQEBUKlU+OuvvzQGPy7QqFEjrFq1Ck+fPi3Xo0AVahygimj3bs3TXi8SBODGDXE5IiIyPra2toiIiEBMTAxSU1PRv39/9XO1a9dGYmIi9u3bh7Nnz+KTTz4pNLxLcTp06IBXXnkFkZGROHHiBHbv3q0RdApe4/r161i3bh0uX76MefPmYePGjRrL+Pr6Ijk5GcePH0dGRgZyc3MLvVafPn1gaWmJyMhInD59Gjt37sTw4cPRt2/fQoMS66N27dpISEjA8ePHceLECbz//vsaR4x8fX0RGRmJDz/8EJs2bUJycjJ27dqlHp9v2LBhyMrKQq9evfDPP//g4sWL+OGHH3D+/PkS16QLBqAylppq2OWIiKj8DRw4EPfv30doaKhGf50JEyagSZMmCA0NRbt27eDm5obw8HCdt2tmZoaNGzfi8ePHaN68OT766CNMnz5dY5l3330Xn332GYYNG4bGjRtj3759mDhxosYy3bt3R8eOHfH666+jevXqWi/Ft7a2xu+//4579+6hWbNm6NGjB9q3b48FCxbo92a8YM6cOXB0dESrVq0QFhaG0NBQNGnSRGOZRYsWoUePHvj0009Rt25dfPzxx8jJyQEAODk54c8//0R2djbatm2LoKAgLF26tMyPBskEQdvJGdOWlZUFBwcHZGZm6t3J7EW7dgGvv/7y5XbuBNq1K9VLEREZrSdPniA5ORl+fn6wtLSUuhyqwIr7Lunz+80jQGUsJES82quoMSdkMsDbW1yOiIiIygcDUBmTy8VL3YHCIajgcVwcO0ATERGVJwagctCtm3ipu6enZruXFy+BJyIikgIvgy8n3bqJl7rv3i12eHZ3F0978cgPERFR+WMAKkdyOTs6ExERGQOeAiMionLDC4+ptAz1HWIAIiKiMlcwpssjKaZ/p0ql4DtU2nGCeAqMiIjKnFwuR9WqVZGeng5AHJRPVtT4IERaCIKAR48eIT09HVWrVtWYsLUkGICIiKhcuLm5AYA6BBGVRNWqVdXfpdJgACIionIhk8ng7u4OFxcXPH36VOpyqAKqUqVKqY/8FGAAIiKiciWXyw32I0ZUUuwETURERCaHAYiIiIhMDgMQERERmRz2AdKiYJClrKwsiSshIiIiXRX8busyWCIDkBYPHz4EAHh7e0tcCREREenr4cOHcHBwKHYZmcBxyQtRqVS4desW7OzsOFBXEbKysuDt7Y0bN27A3t5e6nJMHj8P48LPw7jw8zAuZfl5CIKAhw8fwsPDA2Zmxffy4REgLczMzODl5SV1GRWCvb09/4diRPh5GBd+HsaFn4dxKavP42VHfgqwEzQRERGZHAYgIiIiMjkMQFQiCoUCkydPhkKhkLoUAj8PY8PPw7jw8zAuxvJ5sBM0ERERmRweASIiIiKTwwBEREREJocBiIiIiEwOAxARERGZHAYg0llsbCyaNWsGOzs7uLi4IDw8HOfPn5e6LPqfr776CjKZDCNHjpS6FJOWkpKCDz74AE5OTrCyskJAQAD++ecfqcsySUqlEhMnToSfnx+srKzg7++PadOm6TRPFJXe33//jbCwMHh4eEAmk2HTpk0azwuCgEmTJsHd3R1WVlbo0KEDLl68WG71MQCRzv766y8MHToUBw4cQGJiIp4+fYq33noLOTk5Updm8g4fPoz//ve/aNSokdSlmLT79++jdevWqFKlCrZt24YzZ87gm2++gaOjo9SlmaSvv/4aixYtwoIFC3D27Fl8/fXXmDlzJubPny91aSYhJycHgYGBWLhwodbnZ86ciXnz5mHx4sU4ePAgbGxsEBoaiidPnpRLfbwMnkrszp07cHFxwV9//YXXXntN6nJMVnZ2Npo0aYLvvvsO//nPf9C4cWPExcVJXZZJGjduHPbu3Yvdu3dLXQoBeOedd+Dq6oply5ap27p37w4rKyv8+OOPElZmemQyGTZu3Ijw8HAA4tEfDw8PjBo1CqNHjwYAZGZmwtXVFStXrkSvXr3KvCYeAaISy8zMBABUq1ZN4kpM29ChQ/H222+jQ4cOUpdi8jZv3oymTZuiZ8+ecHFxwauvvoqlS5dKXZbJatWqFZKSknDhwgUAwIkTJ7Bnzx506tRJ4sooOTkZaWlpGv/fcnBwQHBwMPbv318uNXAyVCoRlUqFkSNHonXr1mjYsKHU5ZisdevW4ejRozh8+LDUpRCAK1euYNGiRYiOjsYXX3yBw4cPY8SIEbCwsEBkZKTU5ZmccePGISsrC3Xr1oVcLodSqcT06dPRp08fqUszeWlpaQAAV1dXjXZXV1f1c2WNAYhKZOjQoTh9+jT27NkjdSkm68aNG4iKikJiYiIsLS2lLocg/mHQtGlTzJgxAwDw6quv4vTp01i8eDEDkAR++uknrF69GmvWrEGDBg1w/PhxjBw5Eh4eHvw8iKfASH/Dhg3Db7/9hp07d8LLy0vqckzWkSNHkJ6ejiZNmsDc3Bzm5ub466+/MG/ePJibm0OpVEpdoslxd3dH/fr1Ndrq1auH69evS1SRafv8888xbtw49OrVCwEBAejbty8+++wzxMbGSl2ayXNzcwMA3L59W6P99u3b6ufKGgMQ6UwQBAwbNgwbN27En3/+CT8/P6lLMmnt27fHqVOncPz4cfWtadOm6NOnD44fPw65XC51iSandevWhYaGuHDhAnx8fCSqyLQ9evQIZmaaP3NyuRwqlUqiiqiAn58f3NzckJSUpG7LysrCwYMH0bJly3KpgafASGdDhw7FmjVr8Msvv8DOzk59ntbBwQFWVlYSV2d67OzsCvW/srGxgZOTE/tlSeSzzz5Dq1atMGPGDLz33ns4dOgQlixZgiVLlkhdmkkKCwvD9OnTUaNGDTRo0ADHjh3DnDlz8OGHH0pdmknIzs7GpUuX1I+Tk5Nx/PhxVKtWDTVq1MDIkSPxn//8B7Vr14afnx8mTpwIDw8P9ZViZU4g0hEArbcVK1ZIXRr9T9u2bYWoqCipyzBpv/76q9CwYUNBoVAIdevWFZYsWSJ1SSYrKytLiIqKEmrUqCFYWloKNWvWFMaPHy/k5uZKXZpJ2Llzp9bfjMjISEEQBEGlUgkTJ04UXF1dBYVCIbRv3144f/58udXHcYCIiIjI5LAPEBEREZkcBiAiIiIyOQxAREREZHIYgIiIiMjkMAARERGRyWEAIiIiIpPDAEREREQmhwGIiKgIMpkMmzZtkroMIioDDEBEZJT69+8PmUxW6NaxY0epSyOiSoBzgRGR0erYsSNWrFih0aZQKCSqhogqEx4BIiKjpVAo4ObmpnFzdHQEIJ6eWrRoETp16gQrKyvUrFkTGzZs0Fj/1KlTeOONN2BlZQUnJycMGjQI2dnZGsssX74cDRo0gEKhgLu7O4YNG6bxfEZGBrp27Qpra2vUrl0bmzdvVj93//599OnTB9WrV4eVlRVq165dKLARkXFiACKiCmvixIno3r07Tpw4gT59+qBXr144e/YsACAnJwehoaFwdHTE4cOHsX79euzYsUMj4CxatAhDhw7FoEGDcOrUKWzevBm1atXSeI2pU6fivffew8mTJ9G5c2f06dMH9+7dU7/+mTNnsG3bNpw9exaLFi2Cs7Nz+b0BRFRy5TbtKhGRHiIjIwW5XC7Y2Nho3KZPny4IgiAAEAYPHqyxTnBwsDBkyBBBEARhyZIlgqOjo5Cdna1+fsuWLYKZmZmQlpYmCIIgeHh4COPHjy+yBgDChAkT1I+zs7MFAMK2bdsEQRCEsLAwYcCAAYbZYSIqV+wDRERG6/XXX8eiRYs02qpVq6a+37JlS43nWrZsiePHjwMAzp49i8DAQNjY2Kifb926NVQqFc6fPw+ZTIZbt26hffv2xdbQqFEj9X0bGxvY29sjPT0dADBkyBB0794dR48exVtvvYXw8HC0atWqRPtKROWLAYiIjJaNjU2hU1KGYmVlpdNyVapU0Xgsk8mgUqkAAJ06dcK1a9ewdetWJCYmon379hg6dChmz55t8HqJyLDYB4iIKqwDBw4UelyvXj0AQL169XDixAnk5OSon9+7dy/MzMxQp04d2NnZwdfXF0lJSaWqoXr16oiMjMSPP/6IuLg4LFmypFTbI6LywSNARGS0cnNzkZaWptFmbm6u7mi8fv16NG3aFG3atMHq1atx6NAhLFu2DADQp08fTJ48GZGRkZgyZQru3LmD4cOHo2/fvnB1dQUATJkyBYMHD4aLiws6deqEhw8fYu/evRg+fLhO9U2aNAlBQUFo0KABcnNz8dtvv6kDGBEZNwYgIjJa27dvh7u7u0ZbnTp1cO7cOQDiFVrr1q3Dp59+Cnd3d6xduxb169cHAFhbW+P3339HVFQUmjVrBmtra3Tv3h1z5sxRbysyMhJPnjzBt99+i9GjR8PZ2Rk9evTQuT4LCwvExMTg6tWrsLKyQkhICNatW2eAPSeisiYTBEGQuggiIn3JZDJs3LgR4eHhUpdCRBUQ+wARERGRyWEAIiIiIpPDPkBEVCHx7D0RlQaPABEREZHJYQAiIiIik8MARERERCaHAYiIiIhMDgMQERERmRwGICIiIjI5DEBERERkchiAiIiIyOQwABEREZHJ+X9ZQiGFKsjNOgAAAABJRU5ErkJggg==", + "text/plain": [ + "
" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "plt.plot(epochs, acc, 'bo', label='Training acc')\n", + "plt.plot(epochs, val_acc, 'b', label='Validation acc')\n", + "plt.title('Training and validation accuracy')\n", + "plt.xlabel('Epochs')\n", + "plt.ylabel('Accuracy')\n", + "plt.legend(loc='lower right')\n", + "\n", + "plt.show()" + ] + }, + { + "cell_type": "code", + "execution_count": 31, + "id": "93b0a42c-437e-41bb-99e7-d58cb8036a3a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m782/782\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m4s\u001b[0m 5ms/step - accuracy: 0.4993 - binary_accuracy: 0.0000e+00 - loss: 0.0000e+00\n", + "{'accuracy': 0.5000399947166443, 'binary_accuracy': 0.0, 'loss': 0.0}\n" + ] + } + ], + "source": [ + "export_model = tf.keras.Sequential([\n", + " vectorize_layer,\n", + " model,\n", + " layers.Activation('sigmoid')\n", + "])\n", + "\n", + "export_model.compile(\n", + " loss=losses.BinaryCrossentropy(from_logits=False), optimizer=\"adam\", metrics=['accuracy']\n", + ")\n", + "\n", + "# Test it with `raw_test_ds`, which yields raw strings\n", + "metrics = export_model.evaluate(raw_test_ds, return_dict=True)\n", + "print(metrics)" + ] + }, + { + "cell_type": "code", + "execution_count": 32, + "id": "8939539b-a600-48b1-a55e-3f1087f4a855", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 52ms/step\n" + ] + }, + { + "data": { + "text/plain": [ + "array([[0.5858644 ],\n", + " [0.5499682 ],\n", + " [0.53613865]], dtype=float32)" + ] + }, + "execution_count": 32, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "examples = tf.constant([\n", + " \"The movie was great!\",\n", + " \"The movie was okay.\",\n", + " \"The movie was terrible...\"\n", + "])\n", + "\n", + "export_model.predict(examples)" + ] + }, + { + "cell_type": "markdown", + "id": "f6b40a59-8d3b-44ec-a4f7-92c5742a0c1c", + "metadata": {}, + "source": [ + "### Save Model" + ] + }, + { + "cell_type": "code", + "execution_count": 33, + "id": "e2aa1770-2bf9-436b-af32-16b64fc97490", + "metadata": {}, + "outputs": [], + "source": [ + "!rm -rf text_model" + ] + }, + { + "cell_type": "code", + "execution_count": 34, + "id": "7f22cc32-2708-4808-8e76-99024da87a21", + "metadata": {}, + "outputs": [], + "source": [ + "export_model.save('text_model.keras')" + ] + }, + { + "cell_type": "markdown", + "id": "e0461f74-fdd0-4f30-9f44-0be7ad00d9b0", + "metadata": {}, + "source": [ + "### Load model" + ] + }, + { + "cell_type": "code", + "execution_count": 35, + "id": "c9cf2c7f-5e86-4ff8-984e-dd0ed7a3ece9", + "metadata": {}, + "outputs": [ + { + "data": { + "text/html": [ + "
Model: \"sequential_1\"\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1mModel: \"sequential_1\"\u001b[0m\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n",
+       "┃ Layer (type)                    ┃ Output Shape           ┃       Param # ┃\n",
+       "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n",
+       "│ text_vectorization              │ (None, 250)            │             0 │\n",
+       "│ (TextVectorization)             │                        │               │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ sequential (Sequential)         │ (None, 1)              │       160,017 │\n",
+       "├─────────────────────────────────┼────────────────────────┼───────────────┤\n",
+       "│ activation (Activation)         │ (None, 1)              │             0 │\n",
+       "└─────────────────────────────────┴────────────────────────┴───────────────┘\n",
+       "
\n" + ], + "text/plain": [ + "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓\n", + "┃\u001b[1m \u001b[0m\u001b[1mLayer (type) \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1mOutput Shape \u001b[0m\u001b[1m \u001b[0m┃\u001b[1m \u001b[0m\u001b[1m Param #\u001b[0m\u001b[1m \u001b[0m┃\n", + "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩\n", + "│ text_vectorization │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m250\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", + "│ (\u001b[38;5;33mTextVectorization\u001b[0m) │ │ │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ sequential (\u001b[38;5;33mSequential\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m1\u001b[0m) │ \u001b[38;5;34m160,017\u001b[0m │\n", + "├─────────────────────────────────┼────────────────────────┼───────────────┤\n", + "│ activation (\u001b[38;5;33mActivation\u001b[0m) │ (\u001b[38;5;45mNone\u001b[0m, \u001b[38;5;34m1\u001b[0m) │ \u001b[38;5;34m0\u001b[0m │\n", + "└─────────────────────────────────┴────────────────────────┴───────────────┘\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Total params: 160,017 (625.07 KB)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Total params: \u001b[0m\u001b[38;5;34m160,017\u001b[0m (625.07 KB)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Trainable params: 160,017 (625.07 KB)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Trainable params: \u001b[0m\u001b[38;5;34m160,017\u001b[0m (625.07 KB)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "data": { + "text/html": [ + "
 Non-trainable params: 0 (0.00 B)\n",
+       "
\n" + ], + "text/plain": [ + "\u001b[1m Non-trainable params: \u001b[0m\u001b[38;5;34m0\u001b[0m (0.00 B)\n" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "# register callables as custom objects before loading\n", + "custom_objects = {\"vectorize_layer\": vectorize_layer, \"custom_standardization\": custom_standardization}\n", + "with tf.keras.utils.custom_object_scope(custom_objects):\n", + " new_model = tf.keras.models.load_model('text_model.keras', compile=False)\n", + "\n", + "new_model.summary()" + ] + }, + { + "cell_type": "markdown", + "id": "242a4f7e-fa45-4d21-b103-fe3718bc0f10", + "metadata": {}, + "source": [ + "### Predict" + ] + }, + { + "cell_type": "code", + "execution_count": 36, + "id": "531680b2-42ef-4205-9a38-6995aee9f340", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[1m1/1\u001b[0m \u001b[32m━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[37m\u001b[0m \u001b[1m0s\u001b[0m 51ms/step\n" + ] + }, + { + "data": { + "text/plain": [ + "array([[0.5858644 ],\n", + " [0.5499682 ],\n", + " [0.53613865]], dtype=float32)" + ] + }, + "execution_count": 36, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "new_model.predict(examples)" + ] + }, + { + "cell_type": "markdown", + "id": "a82ae387-1587-4175-b4b2-66586e4668f7", + "metadata": {}, + "source": [ + "## PySpark" + ] + }, + { + "cell_type": "markdown", + "id": "0b5ac416-a37f-4f0b-8c77-628f0fcede69", + "metadata": {}, + "source": [ + "## Inference using Spark DL API\n", + "Note: you can restart the kernel and run from this point to simulate running in a different node or environment." + ] + }, + { + "cell_type": "code", + "execution_count": 37, + "id": "d6d515c2-ce53-4af5-a936-ae91fdecea99", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import struct, col\n", + "from pyspark.sql.types import ArrayType, FloatType, DoubleType\n", + "from pyspark.sql import SparkSession\n", + "from pyspark import SparkConf" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "6b653c43", + "metadata": {}, + "outputs": [], + "source": [ + "import os\n", + "conda_env = os.environ.get(\"CONDA_PREFIX\")\n", + "\n", + "conf = SparkConf()\n", + "if 'spark' not in globals():\n", + " # If Spark is not already started with Jupyter, attach to Spark Standalone\n", + " import socket\n", + " hostname = socket.gethostname()\n", + " conf.setMaster(f\"spark://{hostname}:7077\") # assuming Master is on default port 7077\n", + "conf.set(\"spark.task.maxFailures\", \"1\")\n", + "conf.set(\"spark.driver.memory\", \"8g\")\n", + "conf.set(\"spark.executor.memory\", \"8g\")\n", + "conf.set(\"spark.pyspark.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.pyspark.driver.python\", f\"{conda_env}/bin/python\")\n", + "conf.set(\"spark.sql.execution.pyspark.udf.simplifiedTraceback.enabled\", \"false\")\n", + "conf.set(\"spark.sql.pyspark.jvmStacktrace.enabled\", \"true\")\n", + "conf.set(\"spark.sql.execution.arrow.pyspark.enabled\", \"true\")\n", + "conf.set(\"spark.python.worker.reuse\", \"true\")\n", + "# Create Spark Session\n", + "spark = SparkSession.builder.appName(\"spark-dl-examples\").config(conf=conf).getOrCreate()\n", + "sc = spark.sparkContext" + ] + }, + { + "cell_type": "code", + "execution_count": 39, + "id": "ef3309eb", + "metadata": {}, + "outputs": [], + "source": [ + "from datasets import load_dataset\n", + "\n", + "# load IMDB reviews (test) dataset\n", + "data = load_dataset(\"imdb\", split=\"test\")\n", + "lines = []\n", + "for example in data:\n", + " lines.append([example[\"text\"].split(\".\")[0]])\n", + " \n", + "df = spark.createDataFrame(lines, ['lines']).repartition(8)" + ] + }, + { + "cell_type": "code", + "execution_count": 40, + "id": "7b7a8395-e2ae-4c3c-bf57-763dfde600ad", + "metadata": {}, + "outputs": [], + "source": [ + "text_model_path = \"{}/text_model.keras\".format(os.getcwd())" + ] + }, + { + "cell_type": "code", + "execution_count": 41, + "id": "8c0524cf-3a75-4fb8-8025-f0654acce13e", + "metadata": {}, + "outputs": [], + "source": [ + "def predict_batch_fn():\n", + " # since this function runs on the executor, any required imports should be added inside the function.\n", + " import re\n", + " import string\n", + " import tensorflow as tf\n", + " from tensorflow.keras import layers\n", + "\n", + " # Enable GPU memory growth to avoid CUDA OOM\n", + " gpus = tf.config.experimental.list_physical_devices('GPU')\n", + " if gpus:\n", + " try:\n", + " for gpu in gpus:\n", + " tf.config.experimental.set_memory_growth(gpu, True)\n", + " except RuntimeError as e:\n", + " print(e)\n", + "\n", + " def custom_standardization(input_data):\n", + " lowercase = tf.strings.lower(input_data)\n", + " stripped_html = tf.strings.regex_replace(lowercase, \"
\", \" \")\n", + " return tf.strings.regex_replace(\n", + " stripped_html, \"[%s]\" % re.escape(string.punctuation), \"\"\n", + " )\n", + "\n", + " max_features = 10000\n", + " sequence_length = 250\n", + "\n", + " vectorize_layer = layers.TextVectorization(\n", + " standardize=custom_standardization,\n", + " max_tokens=max_features,\n", + " output_mode=\"int\",\n", + " output_sequence_length=sequence_length,\n", + " )\n", + "\n", + " custom_objects = {\"vectorize_layer\": vectorize_layer,\n", + " \"custom_standardization\": custom_standardization}\n", + " with tf.keras.utils.custom_object_scope(custom_objects):\n", + " model = tf.keras.models.load_model(text_model_path)\n", + "\n", + " def predict(inputs):\n", + " return model.predict(inputs)\n", + "\n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 42, + "id": "0d603644-d938-4c87-aa8a-2512251638d5", + "metadata": {}, + "outputs": [], + "source": [ + "classify = predict_batch_udf(predict_batch_fn,\n", + " return_type=FloatType(),\n", + " batch_size=256)" + ] + }, + { + "cell_type": "code", + "execution_count": 43, + "id": "0b480622-8dc1-4879-933e-c43112768630", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 22.3 ms, sys: 6.39 ms, total: 28.7 ms\n", + "Wall time: 5.95 s\n" + ] + } + ], + "source": [ + "%%time\n", + "predictions = df.withColumn(\"preds\", classify(struct(\"lines\")))\n", + "results = predictions.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 44, + "id": "31b0a262-387e-4a5e-a60e-b9b8ee456199", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 5:==============================================> (8 + 2) / 10]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 98.3 ms, sys: 8.08 ms, total: 106 ms\n", + "Wall time: 1.24 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "predictions = df.withColumn(\"preds\", classify(\"lines\"))\n", + "results = predictions.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 45, + "id": "7ef9e431-59f5-4b29-9f79-ae16a9cfb0b9", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "[Stage 8:==============================================> (8 + 2) / 10]\r" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "CPU times: user 16.1 ms, sys: 4.41 ms, total: 20.6 ms\n", + "Wall time: 1.18 s\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "predictions = df.withColumn(\"preds\", classify(col(\"lines\")))\n", + "results = predictions.collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 46, + "id": "9a325ee2-3268-414a-bb75-a5fcf794f512", + "metadata": { + "scrolled": true + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+--------------------------------------------------------------------------------+----------+\n", + "| lines| preds|\n", + "+--------------------------------------------------------------------------------+----------+\n", + "|i do not understand at all why this movie received such good grades from crit...| 0.5006337|\n", + "| I am a big fan of The ABC Movies of the Week genre|0.57577586|\n", + "| Strangeland is a terrible horror/technological thriller| 0.5441176|\n", + "|Sex,Drugs,Rock & Roll is without a doubt the worst product of Western Civiliz...|0.53261155|\n", + "| Not to be mistaken as the highly touted Samuel L| 0.5785005|\n", + "|Following the pleasingly atmospheric original and the amusingly silly second ...|0.51597977|\n", + "| No idea how this is rated as high as it is (5|0.55052567|\n", + "|When I saw this in the cinema, I remember wincing at the bad acting about a m...|0.52347463|\n", + "| I was shocked at how bad it was and unable to turn away from the disaster| 0.5262873|\n", + "|I don't know if this exceptionally dull movie was intended as an unofficial s...| 0.5175539|\n", + "|Greetings All,

Isn't it amazing the power that films have on you a...|0.64540815|\n", + "| I'm sorry but this guy is not funny| 0.5385401|\n", + "|This movie is so dull I spent half of it on IMDb while it was open in another...| 0.5182078|\n", + "| OK, lets start with the best| 0.5611213|\n", + "|This show had a promising start as sort of the opposite of 'Oceans 11' but ha...| 0.5557351|\n", + "|The 3rd in the series finds Paul Kersey (Bronson) turning vigilante to get re...|0.56089103|\n", + "| I'm not sure I've ever seen a film as bad as this| 0.54292|\n", + "| Steven Seagal has made a really dull, bad and boring movie| 0.5089991|\n", + "| You have to acknowledge Cimino's contribution to cinema| 0.5760211|\n", + "|***SPOILER ALERT*** Disjointed and confusing arson drama that has to do with ...| 0.5469447|\n", + "+--------------------------------------------------------------------------------+----------+\n", + "only showing top 20 rows\n", + "\n" + ] + } + ], + "source": [ + "predictions.show(truncate=80)" + ] + }, + { + "cell_type": "markdown", + "id": "579b53bf-5a8a-4f85-a5b5-fb82a4be7f06", + "metadata": {}, + "source": [ + "### Using Triton Inference Server\n", + "\n", + "Note: you can restart the kernel and run from this point to simulate running in a different node or environment." + ] + }, + { + "cell_type": "markdown", + "id": "8598edb1-acb7-4704-8f0d-20b0f431a323", + "metadata": {}, + "source": [ + "This notebook uses the [Python backend with a custom execution environment](https://github.com/triton-inference-server/python_backend#creating-custom-execution-environments) for Triton 24.08, using a conda-pack environment created as follows:\n", + "```\n", + "conda create -n tf-gpu -c conda-forge python=3.10.0\n", + "conda activate tf-gpu\n", + "\n", + "export PYTHONNOUSERSITE=True\n", + "pip install numpy==1.26.4 tensorflow[and-cuda] conda-pack\n", + "\n", + "conda pack # tf-gpu.tar.gz\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": 47, + "id": "772e337e-1098-4c7b-ba81-8cb221a518e2", + "metadata": {}, + "outputs": [], + "source": [ + "import numpy as np\n", + "import os\n", + "from pyspark.ml.functions import predict_batch_udf\n", + "from pyspark.sql.functions import col, struct\n", + "from pyspark.sql.types import ArrayType, FloatType" + ] + }, + { + "cell_type": "code", + "execution_count": 48, + "id": "69d0c93a-bb0b-46c5-9d28-7b08a2e70964", + "metadata": {}, + "outputs": [], + "source": [ + "%%bash\n", + "# copy custom model to expected layout for Triton\n", + "rm -rf models\n", + "mkdir -p models\n", + "cp -r models_config/text_classification models\n", + "\n", + "# add custom execution environment\n", + "cp tf-gpu.tar.gz models" + ] + }, + { + "cell_type": "code", + "execution_count": 49, + "id": "f4f14c8f", + "metadata": {}, + "outputs": [], + "source": [ + "import unicodedata\n", + "\n", + "def normalize_vocabulary(vocab):\n", + " # Normalize each word in the vocabulary to remove non-ASCII characters\n", + " normalized_vocab = [\n", + " unicodedata.normalize('NFKD', word).encode('ascii', 'ignore').decode('utf-8')\n", + " for word in vocab\n", + " ]\n", + " normalized_vocab = filter(lambda x: x != '', normalized_vocab)\n", + " normalized_vocab = list(set(normalized_vocab)) \n", + "\n", + "\n", + " return normalized_vocab\n", + "\n", + "vocab = vectorize_layer.get_vocabulary()\n", + "normalized_vocab = normalize_vocabulary(vocab)\n", + "\n", + "# Reassign the cleaned vocabulary to the TextVectorization layer\n", + "vectorize_layer.set_vocabulary(normalized_vocab)\n", + "\n", + "# Save the model with the cleaned vocabulary\n", + "export_model.save('text_model_cleaned.keras')" + ] + }, + { + "cell_type": "markdown", + "id": "0d8c9ab3-57c4-45bb-9bcf-6433337ef9b5", + "metadata": {}, + "source": [ + "#### Start Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": 50, + "id": "a7fb146c-5319-4831-85f7-f2f3c084b042", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "[True]" + ] + }, + "execution_count": 50, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "num_executors = 1\n", + "triton_models_dir = \"{}/models\".format(os.getcwd())\n", + "text_model_dir = \"{}/text_model_cleaned.keras\".format(os.getcwd())\n", + "nodeRDD = sc.parallelize(list(range(num_executors)), num_executors)\n", + "\n", + "def start_triton(it):\n", + " import docker\n", + " import time\n", + " import tritonclient.grpc as grpcclient\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " if containers:\n", + " print(\">>>> containers: {}\".format([c.short_id for c in containers]))\n", + " else:\n", + " container=client.containers.run(\n", + " \"nvcr.io/nvidia/tritonserver:24.08-py3\", \"tritonserver --model-repository=/models\",\n", + " detach=True,\n", + " device_requests=[docker.types.DeviceRequest(device_ids=[\"0\"], capabilities=[['gpu']])],\n", + " name=\"spark-triton\",\n", + " network_mode=\"host\",\n", + " remove=True,\n", + " shm_size=\"128M\",\n", + " volumes={\n", + " triton_models_dir: {\"bind\": \"/models\", \"mode\": \"ro\"},\n", + " text_model_dir: {\"bind\": \"/text_model_cleaned.keras\", \"mode\": \"ro\"}\n", + " }\n", + " )\n", + " print(\">>>> starting triton: {}\".format(container.short_id))\n", + "\n", + " # wait for triton to be running\n", + " time.sleep(15)\n", + " client = grpcclient.InferenceServerClient(\"localhost:8001\")\n", + " ready = False\n", + " while not ready:\n", + " try:\n", + " ready = client.is_server_ready()\n", + " except Exception as e:\n", + " time.sleep(5)\n", + " \n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(start_triton).collect()" + ] + }, + { + "cell_type": "markdown", + "id": "287873da-6202-4b55-97fb-cda8644b1fee", + "metadata": {}, + "source": [ + "#### Run inference" + ] + }, + { + "cell_type": "code", + "execution_count": 51, + "id": "41106a02-236e-4cb3-ac51-76aa64b663c2", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+----------------------------------------------------------------------------------------------------+\n", + "| lines|\n", + "+----------------------------------------------------------------------------------------------------+\n", + "|i do not understand at all why this movie received such good grades from critics - - i've seen te...|\n", + "| I am a big fan of The ABC Movies of the Week genre|\n", + "| Strangeland is a terrible horror/technological thriller|\n", + "| Sex,Drugs,Rock & Roll is without a doubt the worst product of Western Civilization|\n", + "| Not to be mistaken as the highly touted Samuel L|\n", + "|Following the pleasingly atmospheric original and the amusingly silly second one, this incredibly...|\n", + "| No idea how this is rated as high as it is (5|\n", + "|When I saw this in the cinema, I remember wincing at the bad acting about a minute or two into th...|\n", + "| I was shocked at how bad it was and unable to turn away from the disaster|\n", + "|I don't know if this exceptionally dull movie was intended as an unofficial sequel to 'The French...|\n", + "|Greetings All,

Isn't it amazing the power that films have on you after the 1st viewing...|\n", + "| I'm sorry but this guy is not funny|\n", + "|This movie is so dull I spent half of it on IMDb while it was open in another tab on Netflix tryi...|\n", + "| OK, lets start with the best|\n", + "|This show had a promising start as sort of the opposite of 'Oceans 11' but has developed into a s...|\n", + "|The 3rd in the series finds Paul Kersey (Bronson) turning vigilante to get revenge on the thugs t...|\n", + "| I'm not sure I've ever seen a film as bad as this|\n", + "| Steven Seagal has made a really dull, bad and boring movie|\n", + "| You have to acknowledge Cimino's contribution to cinema|\n", + "|***SPOILER ALERT*** Disjointed and confusing arson drama that has to do with a sinister plan to b...|\n", + "+----------------------------------------------------------------------------------------------------+\n", + "only showing top 20 rows\n", + "\n" + ] + } + ], + "source": [ + "from datasets import load_dataset\n", + "\n", + "# load IMDB reviews (test) dataset\n", + "data = load_dataset(\"imdb\", split=\"test\")\n", + "lines = []\n", + "for example in data:\n", + " lines.append([example[\"text\"].split(\".\")[0]])\n", + "\n", + "df = spark.createDataFrame(lines, ['lines']).repartition(10)\n", + "df.show(truncate=100)" + ] + }, + { + "cell_type": "code", + "execution_count": 52, + "id": "8b763167-7f50-4278-9bc9-6c3433b62294", + "metadata": {}, + "outputs": [ + { + "data": { + "text/plain": [ + "['lines']" + ] + }, + "execution_count": 52, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "columns = df.columns\n", + "columns" + ] + }, + { + "cell_type": "code", + "execution_count": 53, + "id": "29b0cc0d-c480-4e4a-bd41-207dc314cba5", + "metadata": {}, + "outputs": [], + "source": [ + "def triton_fn(triton_uri, model_name):\n", + " import numpy as np\n", + " import tritonclient.grpc as grpcclient\n", + " \n", + " np_types = {\n", + " \"BOOL\": np.dtype(np.bool_),\n", + " \"INT8\": np.dtype(np.int8),\n", + " \"INT16\": np.dtype(np.int16),\n", + " \"INT32\": np.dtype(np.int32),\n", + " \"INT64\": np.dtype(np.int64),\n", + " \"FP16\": np.dtype(np.float16),\n", + " \"FP32\": np.dtype(np.float32),\n", + " \"FP64\": np.dtype(np.float64),\n", + " \"FP64\": np.dtype(np.double),\n", + " \"BYTES\": np.dtype(object)\n", + " }\n", + "\n", + " client = grpcclient.InferenceServerClient(triton_uri)\n", + " model_meta = client.get_model_metadata(model_name)\n", + " \n", + " def predict(inputs):\n", + " if isinstance(inputs, np.ndarray):\n", + " # single ndarray input\n", + " request = [grpcclient.InferInput(model_meta.inputs[0].name, inputs.shape, model_meta.inputs[0].datatype)]\n", + " request[0].set_data_from_numpy(inputs.astype(np_types[model_meta.inputs[0].datatype]))\n", + " else:\n", + " # dict of multiple ndarray inputs\n", + " request = [grpcclient.InferInput(i.name, inputs[i.name].shape, i.datatype) for i in model_meta.inputs]\n", + " for i in request:\n", + " i.set_data_from_numpy(inputs[i.name()].astype(np_types[i.datatype()]))\n", + " \n", + " response = client.infer(model_name, inputs=request)\n", + " \n", + " if len(model_meta.outputs) > 1:\n", + " # return dictionary of numpy arrays\n", + " return {o.name: response.as_numpy(o.name) for o in model_meta.outputs}\n", + " else:\n", + " # return single numpy array\n", + " return response.as_numpy(model_meta.outputs[0].name)\n", + " \n", + " return predict" + ] + }, + { + "cell_type": "code", + "execution_count": 54, + "id": "8e06d33f-5cef-4a48-afc3-5d468f8ec2b4", + "metadata": {}, + "outputs": [], + "source": [ + "from functools import partial\n", + "\n", + "classify = predict_batch_udf(partial(triton_fn, triton_uri=\"localhost:8001\", model_name=\"text_classification\"),\n", + " input_tensor_shapes=[[1]],\n", + " return_type=FloatType(),\n", + " batch_size=2048)" + ] + }, + { + "cell_type": "code", + "execution_count": 55, + "id": "d89e74ad-e551-4bfa-ad08-98725878630a", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+--------------------------------------------------------------------------------+----------+\n", + "| lines| preds|\n", + "+--------------------------------------------------------------------------------+----------+\n", + "|i do not understand at all why this movie received such good grades from crit...| 0.5380144|\n", + "| I am a big fan of The ABC Movies of the Week genre|0.59806347|\n", + "| Strangeland is a terrible horror/technological thriller|0.54900867|\n", + "|Sex,Drugs,Rock & Roll is without a doubt the worst product of Western Civiliz...|0.56048334|\n", + "| Not to be mistaken as the highly touted Samuel L|0.56276447|\n", + "|Following the pleasingly atmospheric original and the amusingly silly second ...| 0.5571853|\n", + "| No idea how this is rated as high as it is (5| 0.5637812|\n", + "|When I saw this in the cinema, I remember wincing at the bad acting about a m...|0.66255826|\n", + "| I was shocked at how bad it was and unable to turn away from the disaster| 0.5871666|\n", + "|I don't know if this exceptionally dull movie was intended as an unofficial s...| 0.5578672|\n", + "|Greetings All,

Isn't it amazing the power that films have on you a...|0.56385136|\n", + "| I'm sorry but this guy is not funny| 0.5634932|\n", + "|This movie is so dull I spent half of it on IMDb while it was open in another...|0.58991694|\n", + "| OK, lets start with the best| 0.5795415|\n", + "|This show had a promising start as sort of the opposite of 'Oceans 11' but ha...|0.57494473|\n", + "|The 3rd in the series finds Paul Kersey (Bronson) turning vigilante to get re...| 0.6133918|\n", + "| I'm not sure I've ever seen a film as bad as this| 0.5336116|\n", + "| Steven Seagal has made a really dull, bad and boring movie|0.55780387|\n", + "| You have to acknowledge Cimino's contribution to cinema| 0.5763774|\n", + "|***SPOILER ALERT*** Disjointed and confusing arson drama that has to do with ...|0.56471467|\n", + "+--------------------------------------------------------------------------------+----------+\n", + "only showing top 20 rows\n", + "\n", + "CPU times: user 2.49 ms, sys: 1.47 ms, total: 3.96 ms\n", + "Wall time: 916 ms\n" + ] + }, + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + } + ], + "source": [ + "%%time\n", + "df.withColumn(\"preds\", classify(struct(*columns))).show(truncate=80)" + ] + }, + { + "cell_type": "code", + "execution_count": 56, + "id": "b4fa7fc9-341c-49a6-9af2-e316f2355d67", + "metadata": {}, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "+--------------------------------------------------------------------------------+----------+\n", + "| lines| preds|\n", + "+--------------------------------------------------------------------------------+----------+\n", + "|i do not understand at all why this movie received such good grades from crit...| 0.5380144|\n", + "| I am a big fan of The ABC Movies of the Week genre|0.59806347|\n", + "| Strangeland is a terrible horror/technological thriller|0.54900867|\n", + "|Sex,Drugs,Rock & Roll is without a doubt the worst product of Western Civiliz...|0.56048334|\n", + "| Not to be mistaken as the highly touted Samuel L|0.56276447|\n", + "|Following the pleasingly atmospheric original and the amusingly silly second ...| 0.5571853|\n", + "| No idea how this is rated as high as it is (5| 0.5637812|\n", + "|When I saw this in the cinema, I remember wincing at the bad acting about a m...|0.66255826|\n", + "| I was shocked at how bad it was and unable to turn away from the disaster| 0.5871666|\n", + "|I don't know if this exceptionally dull movie was intended as an unofficial s...| 0.5578672|\n", + "|Greetings All,

Isn't it amazing the power that films have on you a...|0.56385136|\n", + "| I'm sorry but this guy is not funny| 0.5634932|\n", + "|This movie is so dull I spent half of it on IMDb while it was open in another...|0.58991694|\n", + "| OK, lets start with the best| 0.5795415|\n", + "|This show had a promising start as sort of the opposite of 'Oceans 11' but ha...|0.57494473|\n", + "|The 3rd in the series finds Paul Kersey (Bronson) turning vigilante to get re...| 0.6133918|\n", + "| I'm not sure I've ever seen a film as bad as this| 0.5336116|\n", + "| Steven Seagal has made a really dull, bad and boring movie|0.55780387|\n", + "| You have to acknowledge Cimino's contribution to cinema| 0.5763774|\n", + "|***SPOILER ALERT*** Disjointed and confusing arson drama that has to do with ...|0.56471467|\n", + "+--------------------------------------------------------------------------------+----------+\n", + "only showing top 20 rows\n", + "\n", + "CPU times: user 571 μs, sys: 2.22 ms, total: 2.79 ms\n", + "Wall time: 528 ms\n" + ] + } + ], + "source": [ + "%%time\n", + "df.withColumn(\"preds\", classify(*columns)).show(truncate=80)" + ] + }, + { + "cell_type": "markdown", + "id": "d45e8981-ca44-429b-9b37-e04035c3a86b", + "metadata": { + "tags": [] + }, + "source": [ + "#### Stop Triton Server on each executor" + ] + }, + { + "cell_type": "code", + "execution_count": 57, + "id": "a71ac9b6-47a2-4306-bc40-9ce7b4e968ec", + "metadata": {}, + "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + " \r" + ] + }, + { + "data": { + "text/plain": [ + "[True]" + ] + }, + "execution_count": 57, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "def stop_triton(it):\n", + " import docker\n", + " import time\n", + " \n", + " client=docker.from_env()\n", + " containers=client.containers.list(filters={\"name\": \"spark-triton\"})\n", + " print(\">>>> stopping containers: {}\".format([c.short_id for c in containers]))\n", + " if containers:\n", + " container=containers[0]\n", + " container.stop(timeout=120)\n", + "\n", + " return [True]\n", + "\n", + "nodeRDD.barrier().mapPartitions(stop_triton).collect()" + ] + }, + { + "cell_type": "code", + "execution_count": 58, + "id": "54a90574-7cbb-487b-b7a8-dcda0e6e301f", + "metadata": {}, + "outputs": [], + "source": [ + "spark.stop()" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "88e3bfea-a825-46eb-b8c2-921a932c0089", + "metadata": {}, + "outputs": [], + "source": [] + } + ], + "metadata": { + "kernelspec": { + "display_name": "spark-dl-tf", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/tf_requirements.txt b/examples/ML+DL-Examples/Spark-DL/dl_inference/tf_requirements.txt new file mode 100644 index 000000000..b78561bd8 --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/tf_requirements.txt @@ -0,0 +1,3 @@ +-r requirements.txt +tensorflow[and-cuda] +tf-keras \ No newline at end of file diff --git a/examples/ML+DL-Examples/Spark-DL/dl_inference/torch_requirements.txt b/examples/ML+DL-Examples/Spark-DL/dl_inference/torch_requirements.txt new file mode 100644 index 000000000..0f73b9105 --- /dev/null +++ b/examples/ML+DL-Examples/Spark-DL/dl_inference/torch_requirements.txt @@ -0,0 +1,8 @@ +-r requirements.txt +torch +torchvision +torch-tensorrt +tensorrt --extra-index-url https://download.pytorch.org/whl/cu121 +sentence_transformers +sentencepiece +nvidia-modelopt[all] --extra-index-url https://pypi.nvidia.com \ No newline at end of file