liveness_tensorflow.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tensorflow liveness local & remote training  \n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "This notebook shows how to use the SageMaker Python SDK to run your code in a local container before deploying to SageMaker's managed training or hosting environments.  This can speed up iterative testing and debugging while using the same familiar Python SDK interface.  Just change your estimator's `instance_type` to `local` (or `local_gpu` if you're using an ml.p2 or ml.p3 notebook instance).\n",
    "\n",
    "In order to use this feature, you'll need to install docker-compose (and nvidia-docker if training with a GPU).\n",
    "\n",
    "**Note: you can only run a single local notebook at one time.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "The **SageMaker Python SDK** helps you deploy your models for training and hosting in optimized, productions ready containers in SageMaker. The SageMaker Python SDK is easy to use, modular, extensible and compatible with TensorFlow, MXNet, PyTorch. This tutorial focuses on how to create a convolutional neural network model to train the [NUAA dataset](http://parnec.nuaa.edu.cn/_upload/tpl/02/db/731/template731/pages/xtan/NUAAImposterDB_download.html) using **Tensorflow in local mode**.\n",
    "\n",
    "### Set up the environment\n",
    "\n",
    "This notebook was created and tested on a single ml.p2.xlarge notebook instance.\n",
    "\n",
    "Let's start by specifying:\n",
    "\n",
    "- The S3 bucket and prefix that you want to use for training and model data. This should be within the same region as the Notebook Instance, training, and hosting.\n",
    "- The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these. Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the sagemaker.get_execution_role() with appropriate full IAM role arn string(s)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Requirement already satisfied: awscli in /opt/conda/envs/liveness/lib/python3.6/site-packages (1.22.10)\n",
      "Requirement already satisfied: PyYAML<5.5,>=3.10 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from awscli) (5.4.1)\n",
      "Requirement already satisfied: colorama<0.4.4,>=0.2.5 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from awscli) (0.4.3)\n",
      "Requirement already satisfied: rsa<4.8,>=3.1.2 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from awscli) (4.7)\n",
      "Requirement already satisfied: botocore==1.23.10 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from awscli) (1.23.10)\n",
      "Requirement already satisfied: docutils<0.16,>=0.10 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from awscli) (0.15.2)\n",
      "Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from awscli) (0.5.0)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from botocore==1.23.10->awscli) (1.26.7)\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from botocore==1.23.10->awscli) (0.10.0)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from botocore==1.23.10->awscli) (2.8.1)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.23.10->awscli) (1.15.0)\n",
      "Requirement already satisfied: pyasn1>=0.1.3 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from rsa<4.8,>=3.1.2->awscli) (0.4.8)\n",
      "Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com\n",
      "Requirement already satisfied: boto3 in /opt/conda/envs/liveness/lib/python3.6/site-packages (1.20.10)\n",
      "Requirement already satisfied: sagemaker in /opt/conda/envs/liveness/lib/python3.6/site-packages (2.69.0)\n",
      "Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from boto3) (0.10.0)\n",
      "Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from boto3) (0.5.0)\n",
      "Requirement already satisfied: botocore<1.24.0,>=1.23.10 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from boto3) (1.23.10)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from botocore<1.24.0,>=1.23.10->boto3) (1.26.7)\n",
      "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from botocore<1.24.0,>=1.23.10->boto3) (2.8.1)\n",
      "Requirement already satisfied: six>=1.5 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.24.0,>=1.23.10->boto3) (1.15.0)\n",
      "Requirement already satisfied: protobuf>=3.1 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (3.14.0)\n",
      "Requirement already satisfied: attrs in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (20.3.0)\n",
      "Requirement already satisfied: google-pasta in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (0.2.0)\n",
      "Requirement already satisfied: pathos in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (0.2.8)\n",
      "Requirement already satisfied: importlib-metadata>=1.4.0 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (2.0.0)\n",
      "Requirement already satisfied: numpy>=1.9.0 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (1.19.5)\n",
      "Requirement already satisfied: protobuf3-to-dict>=0.1.5 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (0.1.5)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (20.9)\n",
      "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (1.0.1)\n",
      "Requirement already satisfied: pandas in /opt/conda/envs/liveness/lib/python3.6/site-packages (from sagemaker) (1.1.5)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from importlib-metadata>=1.4.0->sagemaker) (3.4.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from packaging>=20.0->sagemaker) (2.4.7)\n",
      "Requirement already satisfied: pytz>=2017.2 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from pandas->sagemaker) (2021.1)\n",
      "Requirement already satisfied: dill>=0.3.4 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from pathos->sagemaker) (0.3.4)\n",
      "Requirement already satisfied: ppft>=1.6.6.4 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from pathos->sagemaker) (1.6.6.4)\n",
      "Requirement already satisfied: multiprocess>=0.70.12 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from pathos->sagemaker) (0.70.12.2)\n",
      "Requirement already satisfied: pox>=0.3.0 in /opt/conda/envs/liveness/lib/python3.6/site-packages (from pathos->sagemaker) (0.3.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install awscli --upgrade\n",
    "!pip install boto3 sagemaker"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'2.4.1'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "tf.__version__"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Instance type = local_gpu\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import subprocess\n",
    "\n",
    "instance_type = \"local\"\n",
    "\n",
    "try:\n",
    "    if subprocess.call(\"nvidia-smi\") == 0:\n",
    "        ## Set type to GPU if one is present\n",
    "        instance_type = \"local_gpu\"\n",
    "except:\n",
    "    pass\n",
    "\n",
    "print(\"Instance type = \" + instance_type)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Couldn't call 'get_role' to get Role ARN from role name sagemaker_execution to get Role path.\n"
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "role = sagemaker.get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Download the NUAA dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "process_train_data\n",
      "Read preprocessed data from cache file: train.pkl\n",
      "process_test_data\n",
      "Read preprocessed data from cache file: test.pkl\n",
      "There are 2792 training examples \n",
      "There are 699 validation examples\n",
      "There are 9123 testing examples\n",
      "Image data shape is (64, 64)\n",
      "There are 2 classes\n"
     ]
    }
   ],
   "source": [
    "#!pygmentize utils_tf.py\n",
    "%run utils_tf.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Preview"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "# get some random training images\n",
    "#dataiter = iter(trainloader)\n",
    "#images, labels = dataiter.next()\n",
    "\n",
    "# show images\n",
    "#imshow(torchvision.utils.make_grid(images))\n",
    "\n",
    "# print labels\n",
    "#print(' '.join('%9s' % classes[labels[j]] for j in range(4)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train locally\n",
    "We use the ```sagemaker.Session.upload_data``` function to upload our datasets to an S3 location. The return value inputs identifies the location -- we will use this later when we start the training job."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: SM_NUM_GPUS=1\n",
      "env: SM_MODEL_DIR=/tmp/model\n",
      "env: SM_CHANNEL_TRAINING=upload\n",
      "env: SM_CHANNEL_VALIDATION=upload\n",
      "env: AWS_PROFILE=default-api\n",
      "local_gpu arn:aws:iam::481126471388:role/sagemaker_execution\n"
     ]
    }
   ],
   "source": [
    "%env SM_NUM_GPUS=1\n",
    "%env SM_MODEL_DIR=/tmp/model\n",
    "%env SM_CHANNEL_TRAINING=upload\n",
    "%env SM_CHANNEL_VALIDATION=upload\n",
    "%env AWS_PROFILE=default-api\n",
    "#DUMMY_IAM_ROLE = 'arn:aws:iam::111111111111:role/service-role/AmazonSageMaker-ExecutionRole-20200101T000001'\n",
    "print(instance_type, role)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021-11-22 10:39:10.086720: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n",
      "tensorflow version: 2.4.1\n",
      "(2792, 64, 64) (2792,) (699, 64, 64) (699,)\n",
      "channels_last\n",
      "x_train shape: (2792, 64, 64, 1)\n",
      "2792 train samples\n",
      "699 test samples\n",
      "2021-11-22 10:39:12.143099: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set\n",
      "2021-11-22 10:39:12.164771: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1\n",
      "2021-11-22 10:39:12.205238: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero\n",
      "2021-11-22 10:39:12.206080: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: \n",
      "pciBusID: 0000:01:00.0 name: NVIDIA GeForce GTX 1080 Ti computeCapability: 6.1\n",
      "coreClock: 1.6575GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s\n",
      "2021-11-22 10:39:12.206107: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0\n",
      "2021-11-22 10:39:12.375164: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11\n",
      "2021-11-22 10:39:12.375362: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11\n",
      "2021-11-22 10:39:12.393941: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10\n",
      "2021-11-22 10:39:12.402086: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10\n",
      "2021-11-22 10:39:12.402629: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusolver.so.10'; dlerror: libcusolver.so.10: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda-11.2/lib64:/usr/local/cuda-11.2/lib64:/opt/intel/compilers_and_libraries_2020.0.166/linux/mkl/lib/intel64_lin:\n",
      "2021-11-22 10:39:12.426629: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusparse.so.11\n",
      "2021-11-22 10:39:12.428429: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudnn.so.8\n",
      "2021-11-22 10:39:12.428504: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.\n",
      "Skipping registering GPU devices...\n",
      "2021-11-22 10:39:12.430075: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set\n",
      "2021-11-22 10:39:12.430139: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:\n",
      "2021-11-22 10:39:12.430165: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      \n",
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "conv2d (Conv2D)              (None, 64, 64, 18)        180       \n",
      "_________________________________________________________________\n",
      "activation (Activation)      (None, 64, 64, 18)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization (BatchNo (None, 64, 64, 18)        72        \n",
      "_________________________________________________________________\n",
      "conv2d_1 (Conv2D)            (None, 64, 64, 18)        2934      \n",
      "_________________________________________________________________\n",
      "activation_1 (Activation)    (None, 64, 64, 18)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d (MaxPooling2D) (None, 32, 32, 18)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_1 (Batch (None, 32, 32, 18)        72        \n",
      "_________________________________________________________________\n",
      "conv2d_2 (Conv2D)            (None, 32, 32, 36)        5868      \n",
      "_________________________________________________________________\n",
      "activation_2 (Activation)    (None, 32, 32, 36)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_2 (Batch (None, 32, 32, 36)        144       \n",
      "_________________________________________________________________\n",
      "conv2d_3 (Conv2D)            (None, 32, 32, 36)        11700     \n",
      "_________________________________________________________________\n",
      "activation_3 (Activation)    (None, 32, 32, 36)        0         \n",
      "_________________________________________________________________\n",
      "max_pooling2d_1 (MaxPooling2 (None, 16, 16, 36)        0         \n",
      "_________________________________________________________________\n",
      "batch_normalization_3 (Batch (None, 16, 16, 36)        144       \n",
      "_________________________________________________________________\n",
      "flatten (Flatten)            (None, 9216)              0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 128)               1179776   \n",
      "_________________________________________________________________\n",
      "activation_4 (Activation)    (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dropout (Dropout)            (None, 128)               0         \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (None, 2)                 258       \n",
      "_________________________________________________________________\n",
      "activation_5 (Activation)    (None, 2)                 0         \n",
      "=================================================================\n",
      "Total params: 1,201,148\n",
      "Trainable params: 1,200,932\n",
      "Non-trainable params: 216\n",
      "_________________________________________________________________\n",
      "None\n",
      "[INFO] compiling model...\n",
      "/opt/conda/envs/liveness/lib/python3.6/site-packages/tensorflow/python/keras/engine/training.py:1844: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.\n",
      "  warnings.warn('`Model.fit_generator` is deprecated and '\n",
      "2021-11-22 10:39:13.214019: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)\n",
      "2021-11-22 10:39:13.238859: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 3601855000 Hz\n",
      "87/87 [==============================] - 15s 164ms/step - loss: 1.4114 - accuracy: 0.5181 - val_loss: 0.7005 - val_accuracy: 0.4964\n",
      "Validation loss    : 0.7005442380905151\n",
      "Validation accuracy: 0.4964234530925751\n",
      "2021-11-22 10:39:30.199151: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\n"
     ]
    }
   ],
   "source": [
    "!python train.py --epochs 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "s3://sagemaker-us-east-1-481126471388/keras-liveness/training/training.npz\n",
      "s3://sagemaker-us-east-1-481126471388/keras-liveness/validation/validation.npz\n"
     ]
    }
   ],
   "source": [
    "prefix = 'keras-liveness'\n",
    "\n",
    "#training_input_path   = sagemaker_session.upload_data('upload/training.npz', key_prefix=prefix+'/training')\n",
    "#validation_input_path = sagemaker_session.upload_data('upload/validation.npz', key_prefix=prefix+'/validation')\n",
    "training_input_path   = \"s3://sagemaker-us-east-1-481126471388/keras-liveness/training/training.npz\"\n",
    "validation_input_path = \"s3://sagemaker-us-east-1-481126471388/keras-liveness/validation/validation.npz\"\n",
    "print(training_input_path)\n",
    "print(validation_input_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Construct a script for training \n",
    "Here is the full code for the network model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "train_instance_type has been renamed in sagemaker>=2.\n",
      "See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.\n",
      "train_instance_count has been renamed in sagemaker>=2.\n",
      "See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.\n",
      "train_instance_type has been renamed in sagemaker>=2.\n",
      "See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.\n"
     ]
    }
   ],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "tf_estimator = TensorFlow(entry_point='train.py', \n",
    "#                          source_dir = './code',\n",
    "                          role=role,\n",
    "                          train_instance_count=1, \n",
    "#                           train_instance_type='local_gpu',\n",
    "                          train_instance_type='ml.m4.xlarge',\n",
    "                          framework_version='2.4.1',\n",
    "                          py_version='py37',\n",
    "                          script_mode=True,\n",
    "                          hyperparameters={'epochs': 1}\n",
    "                         )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 42 µs, sys: 0 ns, total: 42 µs\n",
      "Wall time: 54.6 µs\n",
      "2021-11-22 07:39:47 Starting - Starting the training job...\n",
      "2021-11-22 07:39:50 Starting - Launching requested ML instancesProfilerReport-1637566786: InProgress\n",
      "......\n",
      "2021-11-22 07:41:19 Starting - Preparing the instances for training.........\n",
      "2021-11-22 07:43:00 Downloading - Downloading input data...\n",
      "2021-11-22 07:43:20 Training - Downloading the training image..\u001b[34m2021-11-22 07:43:40.241243: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:40.255932: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:105] SageMaker Profiler is not enabled. The timeline writer thread will not be started, future recorded events will be dropped.\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:40.545959: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:46,648 sagemaker-training-toolkit INFO     Imported framework sagemaker_tensorflow_container.training\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:46,656 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:47,460 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:47,479 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:47,497 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:47,510 sagemaker-training-toolkit INFO     Invoking user script\u001b[0m\n",
      "\u001b[34mTraining Env:\u001b[0m\n",
      "\u001b[34m{\n",
      "    \"additional_framework_parameters\": {},\n",
      "    \"channel_input_dirs\": {\n",
      "        \"training\": \"/opt/ml/input/data/training\",\n",
      "        \"validation\": \"/opt/ml/input/data/validation\"\n",
      "    },\n",
      "    \"current_host\": \"algo-1\",\n",
      "    \"framework_module\": \"sagemaker_tensorflow_container.training:main\",\n",
      "    \"hosts\": [\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"hyperparameters\": {\n",
      "        \"model_dir\": \"s3://sagemaker-us-east-1-481126471388/tensorflow-training-2021-11-22-07-39-43-845/model\",\n",
      "        \"epochs\": 1\n",
      "    },\n",
      "    \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "    \"input_data_config\": {\n",
      "        \"training\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        },\n",
      "        \"validation\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        }\n",
      "    },\n",
      "    \"input_dir\": \"/opt/ml/input\",\n",
      "    \"is_master\": true,\n",
      "    \"job_name\": \"tensorflow-training-2021-11-22-07-39-43-845\",\n",
      "    \"log_level\": 20,\n",
      "    \"master_hostname\": \"algo-1\",\n",
      "    \"model_dir\": \"/opt/ml/model\",\n",
      "    \"module_dir\": \"s3://sagemaker-us-east-1-481126471388/tensorflow-training-2021-11-22-07-39-43-845/source/sourcedir.tar.gz\",\n",
      "    \"module_name\": \"train\",\n",
      "    \"network_interface_name\": \"eth0\",\n",
      "    \"num_cpus\": 4,\n",
      "    \"num_gpus\": 0,\n",
      "    \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "    \"output_dir\": \"/opt/ml/output\",\n",
      "    \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "    \"resource_config\": {\n",
      "        \"current_host\": \"algo-1\",\n",
      "        \"hosts\": [\n",
      "            \"algo-1\"\n",
      "        ],\n",
      "        \"network_interface_name\": \"eth0\"\n",
      "    },\n",
      "    \"user_entry_point\": \"train.py\"\u001b[0m\n",
      "\u001b[34m}\u001b[0m\n",
      "\u001b[34mEnvironment variables:\u001b[0m\n",
      "\u001b[34mSM_HOSTS=[\"algo-1\"]\u001b[0m\n",
      "\u001b[34mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
      "\u001b[34mSM_HPS={\"epochs\":1,\"model_dir\":\"s3://sagemaker-us-east-1-481126471388/tensorflow-training-2021-11-22-07-39-43-845/model\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ENTRY_POINT=train.py\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_PARAMS={}\u001b[0m\n",
      "\u001b[34mSM_RESOURCE_CONFIG={\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
      "\u001b[34mSM_INPUT_DATA_CONFIG={\"training\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"validation\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}}\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
      "\u001b[34mSM_CHANNELS=[\"training\",\"validation\"]\u001b[0m\n",
      "\u001b[34mSM_CURRENT_HOST=algo-1\u001b[0m\n",
      "\u001b[34mSM_MODULE_NAME=train\u001b[0m\n",
      "\u001b[34mSM_LOG_LEVEL=20\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_MODULE=sagemaker_tensorflow_container.training:main\u001b[0m\n",
      "\u001b[34mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
      "\u001b[34mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
      "\u001b[34mSM_NUM_CPUS=4\u001b[0m\n",
      "\u001b[34mSM_NUM_GPUS=0\u001b[0m\n",
      "\u001b[34mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
      "\u001b[34mSM_MODULE_DIR=s3://sagemaker-us-east-1-481126471388/tensorflow-training-2021-11-22-07-39-43-845/source/sourcedir.tar.gz\u001b[0m\n",
      "\u001b[34mSM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"training\":\"/opt/ml/input/data/training\",\"validation\":\"/opt/ml/input/data/validation\"},\"current_host\":\"algo-1\",\"framework_module\":\"sagemaker_tensorflow_container.training:main\",\"hosts\":[\"algo-1\"],\"hyperparameters\":{\"epochs\":1,\"model_dir\":\"s3://sagemaker-us-east-1-481126471388/tensorflow-training-2021-11-22-07-39-43-845/model\"},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"training\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"},\"validation\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"is_master\":true,\"job_name\":\"tensorflow-training-2021-11-22-07-39-43-845\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-us-east-1-481126471388/tensorflow-training-2021-11-22-07-39-43-845/source/sourcedir.tar.gz\",\"module_name\":\"train\",\"network_interface_name\":\"eth0\",\"num_cpus\":4,\"num_gpus\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_host\":\"algo-1\",\"hosts\":[\"algo-1\"],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"train.py\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ARGS=[\"--epochs\",\"1\",\"--model_dir\",\"s3://sagemaker-us-east-1-481126471388/tensorflow-training-2021-11-22-07-39-43-845/model\"]\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_TRAINING=/opt/ml/input/data/training\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_VALIDATION=/opt/ml/input/data/validation\u001b[0m\n",
      "\u001b[34mSM_HP_MODEL_DIR=s3://sagemaker-us-east-1-481126471388/tensorflow-training-2021-11-22-07-39-43-845/model\u001b[0m\n",
      "\u001b[34mSM_HP_EPOCHS=1\u001b[0m\n",
      "\u001b[34mPYTHONPATH=/opt/ml/code:/usr/local/bin:/usr/local/lib/python37.zip:/usr/local/lib/python3.7:/usr/local/lib/python3.7/lib-dynload:/usr/local/lib/python3.7/site-packages\u001b[0m\n",
      "\u001b[34mInvoking script with the following command:\u001b[0m\n",
      "\u001b[34m/usr/local/bin/python3.7 train.py --epochs 1 --model_dir s3://sagemaker-us-east-1-481126471388/tensorflow-training-2021-11-22-07-39-43-845/model\u001b[0m\n",
      "\u001b[34mtensorflow version: 2.4.1\u001b[0m\n",
      "\u001b[34m(2792, 64, 64) (2792,) (699, 64, 64) (699,)\u001b[0m\n",
      "\u001b[34mchannels_last\u001b[0m\n",
      "\u001b[34mx_train shape: (2792, 64, 64, 1)\u001b[0m\n",
      "\u001b[34m2792 train samples\u001b[0m\n",
      "\u001b[34m699 test samples\u001b[0m\n",
      "\u001b[34m[2021-11-22 07:43:50.318 ip-10-2-124-62.ec2.internal:27 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[2021-11-22 07:43:50.699 ip-10-2-124-62.ec2.internal:27 INFO profiler_config_parser.py:102] User has disabled profiler.\u001b[0m\n",
      "\u001b[34mModel: \"sequential\"\u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mLayer (type)                 Output Shape              Param #   \u001b[0m\n",
      "\u001b[34m=================================================================\u001b[0m\n",
      "\u001b[34mconv2d (Conv2D)              (None, 64, 64, 18)        180       \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mactivation (Activation)      (None, 64, 64, 18)        0         \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mbatch_normalization (BatchNo (None, 64, 64, 18)        72        \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mconv2d_1 (Conv2D)            (None, 64, 64, 18)        2934      \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mactivation_1 (Activation)    (None, 64, 64, 18)        0         \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mmax_pooling2d (MaxPooling2D) (None, 32, 32, 18)        0         \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mbatch_normalization_1 (Batch (None, 32, 32, 18)        72        \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mconv2d_2 (Conv2D)            (None, 32, 32, 36)        5868      \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mactivation_2 (Activation)    (None, 32, 32, 36)        0         \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mbatch_normalization_2 (Batch (None, 32, 32, 36)        144       \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mconv2d_3 (Conv2D)            (None, 32, 32, 36)        11700     \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mactivation_3 (Activation)    (None, 32, 32, 36)        0         \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mmax_pooling2d_1 (MaxPooling2 (None, 16, 16, 36)        0         \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mbatch_normalization_3 (Batch (None, 16, 16, 36)        144       \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mflatten (Flatten)            (None, 9216)              0         \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mdense (Dense)                (None, 128)               1179776   \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mactivation_4 (Activation)    (None, 128)               0         \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mdropout (Dropout)            (None, 128)               0         \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mdense_1 (Dense)              (None, 2)                 258       \u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mactivation_5 (Activation)    (None, 2)                 0         \u001b[0m\n",
      "\u001b[34m=================================================================\u001b[0m\n",
      "\u001b[34mTotal params: 1,201,148\u001b[0m\n",
      "\u001b[34mTrainable params: 1,200,932\u001b[0m\n",
      "\u001b[34mNon-trainable params: 216\u001b[0m\n",
      "\u001b[34m_________________________________________________________________\u001b[0m\n",
      "\u001b[34mNone\u001b[0m\n",
      "\u001b[34m[INFO] compiling model...\u001b[0m\n",
      "\u001b[34m[2021-11-22 07:43:50.997 ip-10-2-124-62.ec2.internal:27 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[2021-11-22 07:43:50.997 ip-10-2-124-62.ec2.internal:27 INFO hook.py:199] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[2021-11-22 07:43:50.998 ip-10-2-124-62.ec2.internal:27 INFO hook.py:253] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[2021-11-22 07:43:50.998 ip-10-2-124-62.ec2.internal:27 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m[2021-11-22 07:43:51.124 ip-10-2-124-62.ec2.internal:27 INFO hook.py:413] Monitoring the collections: metrics, sm_metrics, losses\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "2021-11-22 07:44:00 Training - Training image download completed. Training in progress.\u001b[34m#015 1/87 [..............................] - ETA: 2:26 - loss: 1.2011 - accuracy: 0.4375#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 2/87 [..............................] - ETA: 17s - loss: 1.3982 - accuracy: 0.4219 #010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 3/87 [>.............................] - ETA: 18s - loss: 1.4937 - accuracy: 0.4201#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 4/87 [>.............................] - ETA: 17s - loss: 1.5281 - accuracy: 0.4303#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 5/87 [>.............................] - ETA: 16s - loss: 1.5663 - accuracy: 0.4305#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 6/87 [=>............................] - ETA: 15s - loss: 1.5805 - accuracy: 0.4352#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 7/87 [=>............................] - ETA: 14s - loss: 1.5896 - accuracy: 0.4387#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 8/87 [=>............................] - ETA: 14s - loss: 1.5846 - accuracy: 0.4444#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#015 9/87 [==>...........................] - ETA: 14s - loss: 1.5818 - accuracy: 0.4490#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01510/87 [==>...........................] - ETA: 13s - loss: 1.5725 - accuracy: 0.4544#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01511/87 [==>...........................] - ETA: 14s - loss: 1.5622 - accuracy: 0.4599#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01512/87 [===>..........................] - ETA: 14s - loss: 1.5537 - accuracy: 0.4645#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01513/87 [===>..........................] - ETA: 14s - loss: 1.5490 - accuracy: 0.4676#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01514/87 [===>..........................] - ETA: 13s - loss: 1.5481 - accuracy: 0.4693#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01515/87 [====>.........................] - ETA: 13s - loss: 1.5490 - accuracy: 0.4704#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01516/87 [====>.........................] - ETA: 13s - loss: 1.5481 - accuracy: 0.4717#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01517/87 [====>.........................] - ETA: 12s - loss: 1.5468 - accuracy: 0.4730#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01518/87 [=====>........................] - ETA: 12s - loss: 1.5463 - accuracy: 0.4741#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01519/87 [=====>........................] - ETA: 12s - loss: 1.5445 - accuracy: 0.4756#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01520/87 [=====>........................] - ETA: 12s - loss: 1.5425 - accuracy: 0.4769#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01521/87 [======>.......................] - ETA: 11s - loss: 1.5402 - accuracy: 0.4783#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01522/87 [======>.......................] - ETA: 11s - loss: 1.5378 - accuracy: 0.4797#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01523/87 [======>.......................] - ETA: 11s - loss: 1.5351 - accuracy: 0.4811#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01524/87 [=======>......................] - ETA: 11s - loss: 1.5325 - accuracy: 0.4821#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01525/87 [=======>......................] - ETA: 10s - loss: 1.5298 - accuracy: 0.4828#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01526/87 [=======>......................] - ETA: 10s - loss: 1.5270 - accuracy: 0.4834#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01527/87 [========>.....................] - ETA: 10s - loss: 1.5235 - accuracy: 0.4842#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01528/87 [========>.....................] - ETA: 10s - loss: 1.5199 - accuracy: 0.4850#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01529/87 [========>.....................] - ETA: 10s - loss: 1.5165 - accuracy: 0.4859#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01530/87 [=========>....................] - ETA: 9s - loss: 1.5131 - accuracy: 0.4867 #010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01531/87 [=========>....................] - ETA: 9s - loss: 1.5097 - accuracy: 0.4875#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01532/87 [==========>...................] - ETA: 9s - loss: 1.5067 - accuracy: 0.4881#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01533/87 [==========>...................] - ETA: 9s - loss: 1.5036 - accuracy: 0.4887#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01534/87 [==========>...................] - ETA: 9s - loss: 1.5001 - accuracy: 0.4894#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01535/87 [===========>..................] - ETA: 9s - loss: 1.4966 - accuracy: 0.4901#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01536/87 [===========>..................] - ETA: 8s - loss: 1.4928 - accuracy: 0.4909#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01537/87 [===========>..................] - ETA: 8s - loss: 1.4889 - accuracy: 0.4917#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01538/87 [============>.................] - ETA: 8s - loss: 1.4851 - accuracy: 0.4925#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01539/87 [============>.................] - ETA: 8s - loss: 1.4813 - accuracy: 0.4932#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01540/87 [============>.................] - ETA: 8s - loss: 1.4774 - accuracy: 0.4940#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01541/87 [=============>................] - ETA: 8s - loss: 1.4738 - accuracy: 0.4947#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01542/87 [=============>................] - ETA: 7s - loss: 1.4707 - accuracy: 0.4953#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01543/87 [=============>................] - ETA: 7s - loss: 1.4678 - accuracy: 0.4960#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01544/87 [==============>...............] - ETA: 7s - loss: 1.4651 - accuracy: 0.4965#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01545/87 [==============>...............] - ETA: 7s - loss: 1.4624 - accuracy: 0.4971#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01546/87 [==============>...............] - ETA: 7s - loss: 1.4598 - accuracy: 0.4977#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01547/87 [===============>..............] - ETA: 7s - loss: 1.4572 - accuracy: 0.4984#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01548/87 [===============>..............] - ETA: 6s - loss: 1.4545 - accuracy: 0.4990#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01549/87 [===============>..............] - ETA: 6s - loss: 1.4519 - accuracy: 0.4996#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01550/87 [================>.............] - ETA: 6s - loss: 1.4492 - accuracy: 0.5002#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01551/87 [================>.............] - ETA: 6s - loss: 1.4467 - accuracy: 0.5008#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01552/87 [================>.............] - ETA: 6s - loss: 1.4441 - accuracy: 0.5014#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01553/87 [=================>............] - ETA: 6s - loss: 1.4415 - accuracy: 0.5021#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01554/87 [=================>............] - ETA: 6s - loss: 1.4390 - accuracy: 0.5027#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01555/87 [=================>............] - ETA: 6s - loss: 1.4363 - accuracy: 0.5033#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01556/87 [==================>...........] - ETA: 6s - loss: 1.4337 - accuracy: 0.5039#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01557/87 [==================>...........] - ETA: 5s - loss: 1.4312 - accuracy: 0.5045#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01558/87 [==================>...........] - ETA: 5s - loss: 1.4287 - accuracy: 0.5051#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01559/87 [===================>..........] - ETA: 5s - loss: 1.4261 - accuracy: 0.5057#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01560/87 [===================>..........] - ETA: 5s - loss: 1.4236 - accuracy: 0.5063#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01561/87 [===================>..........] - ETA: 5s - loss: 1.4210 - accuracy: 0.5070#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01562/87 [====================>.........] - ETA: 5s - loss: 1.4184 - accuracy: 0.5076#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01563/87 [====================>.........] - ETA: 5s - loss: 1.4158 - accuracy: 0.5083#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01564/87 [=====================>........] - ETA: 4s - loss: 1.4133 - accuracy: 0.5089#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01565/87 [=====================>........] - ETA: 4s - loss: 1.4110 - accuracy: 0.5094#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01566/87 [=====================>........] - ETA: 4s - loss: 1.4086 - accuracy: 0.5100#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01567/87 [======================>.......] - ETA: 4s - loss: 1.4063 - accuracy: 0.5106#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01568/87 [======================>.......] - ETA: 4s - loss: 1.4039 - accuracy: 0.5111#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01569/87 [======================>.......] - ETA: 3s - loss: 1.4015 - accuracy: 0.5117#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01570/87 [=======================>......] - ETA: 3s - loss: 1.3991 - accuracy: 0.5122#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01571/87 [=======================>......] - ETA: 3s - loss: 1.3967 - accuracy: 0.5128#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01572/87 [=======================>......] - ETA: 3s - loss: 1.3942 - accuracy: 0.5134#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01573/87 [========================>.....] - ETA: 2s - loss: 1.3918 - accuracy: 0.5140#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01574/87 [========================>.....] - ETA: 2s - loss: 1.3893 - accuracy: 0.5146#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01575/87 [========================>.....] - ETA: 2s - loss: 1.3870 - accuracy: 0.5152#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01576/87 [=========================>....] - ETA: 2s - loss: 1.3846 - accuracy: 0.5157#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01577/87 [=========================>....] - ETA: 2s - loss: 1.3824 - accuracy: 0.5163#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01578/87 [=========================>....] - ETA: 1s - loss: 1.3801 - accuracy: 0.5169#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01579/87 [==========================>...] - ETA: 1s - loss: 1.3778 - accuracy: 0.5174#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01580/87 [==========================>...] - ETA: 1s - loss: 1.3756 - accuracy: 0.5180#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01581/87 [==========================>...] - ETA: 1s - loss: 1.3734 - accuracy: 0.5186#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01582/87 [===========================>..] - ETA: 1s - loss: 1.3711 - accuracy: 0.5192#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01583/87 [===========================>..] - ETA: 0s - loss: 1.3689 - accuracy: 0.5197#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01584/87 [===========================>..] - ETA: 0s - loss: 1.3667 - accuracy: 0.5203#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01585/87 [============================>.] - ETA: 0s - loss: 1.3646 - accuracy: 0.5208#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01586/87 [============================>.] - ETA: 0s - loss: 1.3624 - accuracy: 0.5213#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01587/87 [============================>.] - ETA: 0s - loss: 1.3604 - accuracy: 0.5219#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01588/87 [==============================] - ETA: 0s - loss: 1.3584 - accuracy: 0.5224#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#010#01587/87 [==============================] - 21s 219ms/step - loss: 1.3564 - accuracy: 0.5228 - val_loss: 0.7263 - val_accuracy: 0.5107\u001b[0m\n",
      "\u001b[34mValidation loss    : 0.726306676864624\u001b[0m\n",
      "\u001b[34mValidation accuracy: 0.5107296109199524\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:47.664429: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:47.664579: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:105] SageMaker Profiler is not enabled. The timeline writer thread will not be started, future recorded events will be dropped.\u001b[0m\n",
      "\u001b[34m2021-11-22 07:43:47.695053: W tensorflow/core/profiler/internal/smprofiler_timeline.cc:460] Initializing the SageMaker Profiler.\u001b[0m\n",
      "\u001b[34m/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py:1901: UserWarning: `Model.fit_generator` is deprecated and will be removed in a future version. Please use `Model.fit`, which supports generators.\n",
      "  warnings.warn('`Model.fit_generator` is deprecated and '\u001b[0m\n",
      "\u001b[34m2021-11-22 07:44:13.639533: W tensorflow/python/util/util.cc:348] Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.\u001b[0m\n",
      "\u001b[34mINFO:tensorflow:Assets written to: /opt/ml/model/1/assets\u001b[0m\n",
      "\u001b[34mINFO:tensorflow:Assets written to: /opt/ml/model/1/assets\u001b[0m\n",
      "\u001b[34m2021-11-22 07:44:16,028 sagemaker-training-toolkit INFO     Reporting training SUCCESS\u001b[0m\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "2021-11-22 07:44:28 Uploading - Uploading generated training model\n",
      "2021-11-22 07:44:28 Completed - Training job completed\n",
      "Training seconds: 101\n",
      "Billable seconds: 101\n",
      "jupyter.cell:done!\n"
     ]
    }
   ],
   "source": [
    "%time\n",
    "tf_estimator.fit({'training': training_input_path, 'validation': validation_input_path})\n",
    "print (\"jupyter.cell:done!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (liveness)",
   "language": "python",
   "name": "liveness"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.12"
  },
  "notice": "Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License."
 },
 "nbformat": 4,
 "nbformat_minor": 2
}