model_hyper_search.ipynb

{"cells":[{"cell_type":"markdown","id":"JkDEAk-T_HnQ","metadata":{"id":"JkDEAk-T_HnQ"},"source":["**Group 22**\n","\n","Name  | Surname | Email  \n","---------|-------------------|---------\n","Julio|Vigueras|20220661@novaims.unl.pt \n","Ariel|Pérez|20220662@novaims.unl.pt\n","Miguelanguel|Mayuare|20220665@novaims.unl.pt\n","Ayotunde|Aribo|20221012@novaims.unl.pt"]},{"cell_type":"markdown","id":"7ab036f9-bea9-4893-9770-d29fa512eed8","metadata":{"id":"7ab036f9-bea9-4893-9770-d29fa512eed8"},"source":["# Hyper-parameters Tuning\n","-----"]},{"cell_type":"code","execution_count":null,"id":"1c97ffb8-1a1a-46b3-88cf-8df4fd472254","metadata":{"id":"1c97ffb8-1a1a-46b3-88cf-8df4fd472254","tags":[]},"outputs":[],"source":["# Make the imports\n","from tensorflow import keras\n","from tensorflow.keras import layers, initializers\n","from tensorflow.keras.models import load_model\n","from tensorflow.keras.utils import image_dataset_from_directory\n","\n","import plotly.express as px\n","from plotly.subplots import make_subplots\n","import pandas as pd\n","import pathlib\n","\n","import keras_tuner as kt"]},{"cell_type":"markdown","id":"3sXatbSgYjHb","metadata":{"id":"3sXatbSgYjHb"},"source":["For the search of hyper-parameters, bayesian optimization was used\n","\n","Bayesian optimization is a technique for optimizing hyperparameters of a CNN by modeling the objective function as a Gaussian process. It samples new hyperparameters based on this model to guide the search for the optimal values. It balances exploration and exploitation of the hyperparameter space effectively and can lead to finding better hyperparameters in fewer iterations.\n","\n","*Snoek, J., Larochelle, H., & Adams, R. P. (2012). Practical bayesian optimization of machine learning algorithms. In Advances in neural information processing systems (pp. 2951-2959).*\n","\n","While keras tuner provides several algorithms for the task, we had to choose one based on:\n","\n","Bayesian optimization is particularly useful for optimizing hyperparameters of a CNN because it can balance exploration and exploitation of the hyperparameter space effectively, which can lead to finding better hyperparameters in fewer iterations and it can take into account previous observations to update the model of the objective function, making it more efficient and adaptive to the specific problem.\n","\n","The latter statement doesn't mean that other algorithms were worse but under our research and with the limited time for the project we had to chose one.\n","\n","The model tuned, while having the same performance, is less complex and trains faster than D, that makes it a better model."]},{"cell_type":"code","execution_count":null,"id":"7303d1d6-339c-47d1-b098-12ef95fd1b1e","metadata":{"id":"7303d1d6-339c-47d1-b098-12ef95fd1b1e","tags":[]},"outputs":[],"source":["def model_builder(hp):\n","    blocks = hp.Int('blocks', min_value=3, max_value=4, step=1)\n","    data_augmentation = keras.Sequential([\n","        layers.RandomRotation(hp.Float('rotation', min_value=0.05, max_value=0.2, step=0.05)),\n","        layers.RandomFlip(),\n","        layers.RandomContrast(hp.Float('contrast', min_value=0.05, max_value=0.2, step=0.05)),\n","        layers.RandomBrightness(hp.Float('brightness', min_value=0.05, max_value=0.2, step=0.05)),\n","        layers.RandomZoom(hp.Float('zoom', min_value=0, max_value=0.2, step=0.05)),\n","    ])\n","\n","    inputs = keras.Input(shape=(224, 224, 3))\n","    x = data_augmentation(inputs)\n","    x = layers.Rescaling(1./255)(x)\n","    for i in range(blocks + 1, blocks + 5):\n","        x = layers.Conv2D(filters=2**i, kernel_size=3,\n","                          kernel_initializer=initializers.GlorotNormal(seed=123), \n","                          activation=\"relu\")(x)\n","        x = layers.Conv2D(filters=2**i, kernel_size=3, use_bias=False,\n","                          kernel_initializer=initializers.GlorotNormal(seed=123))(x)\n","        x = layers.BatchNormalization()(x)\n","        x = layers.Activation(\"relu\")(x)\n","        x = layers.MaxPooling2D(pool_size=2, strides=2)(x)\n","        x = layers.Dropout(hp.Float('dropout', min_value=0, max_value=0.5, step=0.1))(x)\n","    x = layers.Conv2D(filters=256, kernel_size=3, use_bias=False)(x)\n","    x = layers.BatchNormalization()(x)\n","    x = layers.Activation(\"relu\")(x)\n","    x = layers.Flatten()(x)\n","    x = layers.Dropout(hp.Float('dropout', min_value=0, max_value=0.5, step=0.1))(x)\n","    outputs = layers.Dense(30, activation=\"softmax\")(x)\n","    model = keras.Model(inputs=inputs, outputs=outputs)\n","\n","    learning_rate = hp.Choice('learning_rate', values=[0.0001, 0.001, 0.01])\n","    model.compile(loss=\"sparse_categorical_crossentropy\",\n","              optimizer=keras.optimizers.Adam(learning_rate=learning_rate),\n","              metrics=[\"accuracy\"])\n","    return model\n"]},{"cell_type":"markdown","id":"iJu429xwbKeL","metadata":{"id":"iJu429xwbKeL"},"source":["The next code uses the BayesianOptimization class of the keras-tuner package to fit hyperparameters of a machine learning model.\n","\n","**model_builder** is the function that defines the machine learning model to be tuned. The function takes an argument hp, which is an object used to define the possible values of the hyperparameters and their ranges.\n","\n","**objective** is the objective to be optimized. In this case, you want to minimize the validation loss (val_loss).\n","\n","**max_trials** is the maximum number of times the model is evaluated with different hyperparameter values.\n","\n","**overwrite** is a boolean value indicating whether to overwrite the previous hyperparameter search results or not. If set to True, the previous results will be overwritten."]},{"cell_type":"code","execution_count":null,"id":"0e8a0800-48c6-4538-acf9-5d26f2fc69fa","metadata":{"id":"0e8a0800-48c6-4538-acf9-5d26f2fc69fa","tags":[]},"outputs":[],"source":["tuner = kt.BayesianOptimization(\n","            model_builder,\n","            objective='val_loss',\n","            max_trials=15,\n","            overwrite=True)"]},{"cell_type":"code","execution_count":null,"id":"5qMwm93rZj24","metadata":{"id":"5qMwm93rZj24","tags":[]},"outputs":[],"source":["dataset_path = pathlib.Path(\"moths\")\n","input_shape = (224,224,3)\n","batch_size=64"]},{"cell_type":"code","execution_count":null,"id":"700f43dd-c88c-4bb9-b859-b02f17ead381","metadata":{"id":"700f43dd-c88c-4bb9-b859-b02f17ead381","outputId":"8505b59d-2dd3-4dbc-d5a7-59e2c0966302","tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Found 3558 files belonging to 30 classes.\n","Found 445 files belonging to 30 classes.\n","Found 408 files belonging to 30 classes.\n"]}],"source":["# Split dataset\n","train_dataset = image_dataset_from_directory(\n","    dataset_path / \"train\",\n","    image_size=input_shape[:2],\n","    batch_size=batch_size)\n","validation_dataset = image_dataset_from_directory(\n","    dataset_path / \"valid\",\n","    image_size=input_shape[:2],\n","    batch_size=batch_size)\n","test_dataset = image_dataset_from_directory(\n","    dataset_path / \"test\",\n","    image_size=input_shape[:2],\n","    batch_size=batch_size)"]},{"cell_type":"code","execution_count":null,"id":"3efd84a7-9ff5-492e-914a-40fe2738176f","metadata":{"id":"3efd84a7-9ff5-492e-914a-40fe2738176f","outputId":"04e116a0-bdc2-45bd-df35-b06ebfe4837c","tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Trial 15 Complete [00h 02m 19s]\n","val_loss: 0.7281265258789062\n","\n","Best val_loss So Far: 0.7281265258789062\n","Total elapsed time: 00h 42m 52s\n","INFO:tensorflow:Oracle triggered exit\n"]}],"source":["tuner.search(x=train_dataset, epochs=20, validation_data=validation_dataset)\n","\n","best_model = tuner.get_best_models(num_models=1)[0]\n","best_hyperparameters = tuner.get_best_hyperparameters(num_trials=1)[0]"]},{"cell_type":"code","execution_count":null,"id":"b1deb296-dbd9-4059-92e7-246dc570d006","metadata":{"id":"b1deb296-dbd9-4059-92e7-246dc570d006","outputId":"e6bd61f9-420f-4a07-c400-055c7dd85e66","tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Results summary\n","Results in ./untitled_project\n","Showing 10 best trials\n","Objective(name=\"val_loss\", direction=\"min\")\n","\n","Trial 14 summary\n","Hyperparameters:\n","blocks: 3\n","rotation: 0.1\n","contrast: 0.15000000000000002\n","brightness: 0.15000000000000002\n","zoom: 0.2\n","dropout: 0.2\n","learning_rate: 0.001\n","Score: 0.7281265258789062\n","\n","Trial 10 summary\n","Hyperparameters:\n","blocks: 3\n","rotation: 0.15000000000000002\n","contrast: 0.2\n","brightness: 0.1\n","zoom: 0.1\n","dropout: 0.2\n","learning_rate: 0.001\n","Score: 0.8204528093338013\n","\n","Trial 07 summary\n","Hyperparameters:\n","blocks: 3\n","rotation: 0.1\n","contrast: 0.2\n","brightness: 0.15000000000000002\n","zoom: 0.2\n","dropout: 0.1\n","learning_rate: 0.001\n","Score: 0.8333144187927246\n","\n","Trial 13 summary\n","Hyperparameters:\n","blocks: 4\n","rotation: 0.05\n","contrast: 0.2\n","brightness: 0.2\n","zoom: 0.05\n","dropout: 0.0\n","learning_rate: 0.0001\n","Score: 0.8442155122756958\n","\n","Trial 02 summary\n","Hyperparameters:\n","blocks: 4\n","rotation: 0.05\n","contrast: 0.15000000000000002\n","brightness: 0.1\n","zoom: 0.15000000000000002\n","dropout: 0.2\n","learning_rate: 0.001\n","Score: 0.9749882817268372\n","\n","Trial 08 summary\n","Hyperparameters:\n","blocks: 3\n","rotation: 0.1\n","contrast: 0.2\n","brightness: 0.1\n","zoom: 0.0\n","dropout: 0.0\n","learning_rate: 0.001\n","Score: 1.03951895236969\n","\n","Trial 03 summary\n","Hyperparameters:\n","blocks: 3\n","rotation: 0.15000000000000002\n","contrast: 0.05\n","brightness: 0.15000000000000002\n","zoom: 0.15000000000000002\n","dropout: 0.1\n","learning_rate: 0.0001\n","Score: 1.1061612367630005\n","\n","Trial 06 summary\n","Hyperparameters:\n","blocks: 4\n","rotation: 0.05\n","contrast: 0.2\n","brightness: 0.15000000000000002\n","zoom: 0.0\n","dropout: 0.2\n","learning_rate: 0.0001\n","Score: 1.4825198650360107\n","\n","Trial 04 summary\n","Hyperparameters:\n","blocks: 3\n","rotation: 0.05\n","contrast: 0.05\n","brightness: 0.1\n","zoom: 0.0\n","dropout: 0.4\n","learning_rate: 0.01\n","Score: 1.768874168395996\n","\n","Trial 12 summary\n","Hyperparameters:\n","blocks: 3\n","rotation: 0.1\n","contrast: 0.2\n","brightness: 0.15000000000000002\n","zoom: 0.05\n","dropout: 0.0\n","learning_rate: 0.01\n","Score: 2.0941853523254395\n"]}],"source":["tuner.results_summary()"]},{"cell_type":"code","execution_count":null,"id":"2328667e-bf07-4b9f-b829-604cd0c5ff2e","metadata":{"id":"2328667e-bf07-4b9f-b829-604cd0c5ff2e","outputId":"61465507-3ff1-48a4-ba6d-37385db24094","tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Model: \"model\"\n","_________________________________________________________________\n"," Layer (type)                Output Shape              Param #   \n","=================================================================\n"," input_1 (InputLayer)        [(None, 224, 224, 3)]     0         \n","                                                                 \n"," sequential (Sequential)     (None, 224, 224, 3)       0         \n","                                                                 \n"," rescaling (Rescaling)       (None, 224, 224, 3)       0         \n","                                                                 \n"," conv2d (Conv2D)             (None, 222, 222, 16)      448       \n","                                                                 \n"," conv2d_1 (Conv2D)           (None, 220, 220, 16)      2304      \n","                                                                 \n"," batch_normalization (BatchN  (None, 220, 220, 16)     64        \n"," ormalization)                                                   \n","                                                                 \n"," activation (Activation)     (None, 220, 220, 16)      0         \n","                                                                 \n"," max_pooling2d (MaxPooling2D  (None, 110, 110, 16)     0         \n"," )                                                               \n","                                                                 \n"," dropout (Dropout)           (None, 110, 110, 16)      0         \n","                                                                 \n"," conv2d_2 (Conv2D)           (None, 108, 108, 32)      4640      \n","                                                                 \n"," conv2d_3 (Conv2D)           (None, 106, 106, 32)      9216      \n","                                                                 \n"," batch_normalization_1 (Batc  (None, 106, 106, 32)     128       \n"," hNormalization)                                                 \n","                                                                 \n"," activation_1 (Activation)   (None, 106, 106, 32)      0         \n","                                                                 \n"," max_pooling2d_1 (MaxPooling  (None, 53, 53, 32)       0         \n"," 2D)                                                             \n","                                                                 \n"," dropout_1 (Dropout)         (None, 53, 53, 32)        0         \n","                                                                 \n"," conv2d_4 (Conv2D)           (None, 51, 51, 64)        18496     \n","                                                                 \n"," conv2d_5 (Conv2D)           (None, 49, 49, 64)        36864     \n","                                                                 \n"," batch_normalization_2 (Batc  (None, 49, 49, 64)       256       \n"," hNormalization)                                                 \n","                                                                 \n"," activation_2 (Activation)   (None, 49, 49, 64)        0         \n","                                                                 \n"," max_pooling2d_2 (MaxPooling  (None, 24, 24, 64)       0         \n"," 2D)                                                             \n","                                                                 \n"," dropout_2 (Dropout)         (None, 24, 24, 64)        0         \n","                                                                 \n"," conv2d_6 (Conv2D)           (None, 22, 22, 128)       73856     \n","                                                                 \n"," conv2d_7 (Conv2D)           (None, 20, 20, 128)       147456    \n","                                                                 \n"," batch_normalization_3 (Batc  (None, 20, 20, 128)      512       \n"," hNormalization)                                                 \n","                                                                 \n"," activation_3 (Activation)   (None, 20, 20, 128)       0         \n","                                                                 \n"," max_pooling2d_3 (MaxPooling  (None, 10, 10, 128)      0         \n"," 2D)                                                             \n","                                                                 \n"," dropout_3 (Dropout)         (None, 10, 10, 128)       0         \n","                                                                 \n"," conv2d_8 (Conv2D)           (None, 8, 8, 256)         294912    \n","                                                                 \n"," batch_normalization_4 (Batc  (None, 8, 8, 256)        1024      \n"," hNormalization)                                                 \n","                                                                 \n"," activation_4 (Activation)   (None, 8, 8, 256)         0         \n","                                                                 \n"," flatten (Flatten)           (None, 16384)             0         \n","                                                                 \n"," dropout_4 (Dropout)         (None, 16384)             0         \n","                                                                 \n"," dense (Dense)               (None, 30)                491550    \n","                                                                 \n","=================================================================\n","Total params: 1,081,726\n","Trainable params: 1,080,734\n","Non-trainable params: 992\n","_________________________________________________________________\n"]}],"source":["best_model.build(input_shape=(224, 224, 3))\n","best_model.summary()"]},{"cell_type":"code","execution_count":null,"id":"2693321d-ee26-4521-bdff-d2d41aeb08b9","metadata":{"id":"2693321d-ee26-4521-bdff-d2d41aeb08b9","tags":[]},"outputs":[],"source":["model_best_hps = model_builder(best_hyperparameters)"]},{"cell_type":"code","execution_count":null,"id":"174dfaa6-44d3-4247-9179-12ade31e3b30","metadata":{"id":"174dfaa6-44d3-4247-9179-12ade31e3b30","outputId":"99d736f7-c545-477d-b437-2c63541f586c","tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["Epoch 1/200\n"]},{"name":"stderr","output_type":"stream","text":["2023-04-07 06:37:47.745564: E tensorflow/core/grappler/optimizers/meta_optimizer.cc:954] layout failed: INVALID_ARGUMENT: Size of values 0 does not match size of permutation 4 @ fanin shape inmodel_1/dropout_5/dropout/SelectV2-2-TransposeNHWCToNCHW-LayoutOptimizer\n"]},{"name":"stdout","output_type":"stream","text":["56/56 [==============================] - 9s 124ms/step - loss: 3.6138 - accuracy: 0.2052 - val_loss: 4.9600 - val_accuracy: 0.0427\n","Epoch 2/200\n","56/56 [==============================] - 7s 120ms/step - loss: 2.3612 - accuracy: 0.3623 - val_loss: 8.7740 - val_accuracy: 0.0404\n","Epoch 3/200\n","56/56 [==============================] - 7s 120ms/step - loss: 1.8943 - accuracy: 0.4539 - val_loss: 8.1683 - val_accuracy: 0.0607\n","Epoch 4/200\n","56/56 [==============================] - 7s 121ms/step - loss: 1.6007 - accuracy: 0.5188 - val_loss: 7.0415 - val_accuracy: 0.1461\n","Epoch 5/200\n","56/56 [==============================] - 7s 122ms/step - loss: 1.4437 - accuracy: 0.5658 - val_loss: 4.8755 - val_accuracy: 0.1371\n","Epoch 6/200\n","56/56 [==============================] - 7s 122ms/step - loss: 1.3259 - accuracy: 0.5953 - val_loss: 3.4937 - val_accuracy: 0.2607\n","Epoch 7/200\n","56/56 [==============================] - 7s 122ms/step - loss: 1.2216 - accuracy: 0.6313 - val_loss: 1.6532 - val_accuracy: 0.4899\n","Epoch 8/200\n","56/56 [==============================] - 7s 121ms/step - loss: 1.1205 - accuracy: 0.6509 - val_loss: 1.7588 - val_accuracy: 0.5011\n","Epoch 9/200\n","56/56 [==============================] - 7s 121ms/step - loss: 1.0822 - accuracy: 0.6675 - val_loss: 1.1058 - val_accuracy: 0.6742\n","Epoch 10/200\n","56/56 [==============================] - 7s 122ms/step - loss: 1.0238 - accuracy: 0.6855 - val_loss: 1.3345 - val_accuracy: 0.6270\n","Epoch 11/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.9688 - accuracy: 0.7004 - val_loss: 1.4844 - val_accuracy: 0.5865\n","Epoch 12/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.9022 - accuracy: 0.7167 - val_loss: 1.5157 - val_accuracy: 0.5978\n","Epoch 13/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.8539 - accuracy: 0.7274 - val_loss: 1.1387 - val_accuracy: 0.6674\n","Epoch 14/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.8334 - accuracy: 0.7369 - val_loss: 1.1723 - val_accuracy: 0.6674\n","Epoch 15/200\n","56/56 [==============================] - 7s 123ms/step - loss: 0.7939 - accuracy: 0.7479 - val_loss: 0.7895 - val_accuracy: 0.7618\n","Epoch 16/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.7886 - accuracy: 0.7544 - val_loss: 1.4151 - val_accuracy: 0.6764\n","Epoch 17/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.7328 - accuracy: 0.7622 - val_loss: 0.9418 - val_accuracy: 0.7281\n","Epoch 18/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.7155 - accuracy: 0.7746 - val_loss: 0.7807 - val_accuracy: 0.7775\n","Epoch 19/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.6492 - accuracy: 0.7948 - val_loss: 0.8362 - val_accuracy: 0.7640\n","Epoch 20/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.6150 - accuracy: 0.8058 - val_loss: 0.9926 - val_accuracy: 0.7213\n","Epoch 21/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.6428 - accuracy: 0.7954 - val_loss: 0.8006 - val_accuracy: 0.7708\n","Epoch 22/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.6228 - accuracy: 0.8061 - val_loss: 1.2257 - val_accuracy: 0.6966\n","Epoch 23/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.5712 - accuracy: 0.8210 - val_loss: 0.8753 - val_accuracy: 0.7483\n","Epoch 24/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.5578 - accuracy: 0.8162 - val_loss: 1.3877 - val_accuracy: 0.6584\n","Epoch 25/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.5537 - accuracy: 0.8238 - val_loss: 0.8849 - val_accuracy: 0.7483\n","Epoch 26/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.5207 - accuracy: 0.8283 - val_loss: 0.8334 - val_accuracy: 0.7708\n","Epoch 27/200\n","56/56 [==============================] - 7s 123ms/step - loss: 0.5299 - accuracy: 0.8328 - val_loss: 0.7227 - val_accuracy: 0.8247\n","Epoch 28/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.4846 - accuracy: 0.8435 - val_loss: 0.8948 - val_accuracy: 0.7978\n","Epoch 29/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.4917 - accuracy: 0.8392 - val_loss: 0.9659 - val_accuracy: 0.7258\n","Epoch 30/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.4413 - accuracy: 0.8600 - val_loss: 0.7246 - val_accuracy: 0.8270\n","Epoch 31/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.4780 - accuracy: 0.8454 - val_loss: 0.8857 - val_accuracy: 0.7753\n","Epoch 32/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.4230 - accuracy: 0.8642 - val_loss: 1.0813 - val_accuracy: 0.7416\n","Epoch 33/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.4466 - accuracy: 0.8482 - val_loss: 0.7771 - val_accuracy: 0.8112\n","Epoch 34/200\n","56/56 [==============================] - 7s 123ms/step - loss: 0.4216 - accuracy: 0.8620 - val_loss: 0.5654 - val_accuracy: 0.8449\n","Epoch 35/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.4238 - accuracy: 0.8657 - val_loss: 0.9416 - val_accuracy: 0.7573\n","Epoch 36/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.3853 - accuracy: 0.8749 - val_loss: 0.6464 - val_accuracy: 0.8247\n","Epoch 37/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.4004 - accuracy: 0.8699 - val_loss: 0.9111 - val_accuracy: 0.7596\n","Epoch 38/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.3956 - accuracy: 0.8738 - val_loss: 0.7950 - val_accuracy: 0.7978\n","Epoch 39/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.3808 - accuracy: 0.8766 - val_loss: 0.6971 - val_accuracy: 0.8247\n","Epoch 40/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.3274 - accuracy: 0.8926 - val_loss: 1.2022 - val_accuracy: 0.7191\n","Epoch 41/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.3453 - accuracy: 0.8859 - val_loss: 0.7671 - val_accuracy: 0.8135\n","Epoch 42/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.3491 - accuracy: 0.8865 - val_loss: 0.6675 - val_accuracy: 0.8247\n","Epoch 43/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.3319 - accuracy: 0.8926 - val_loss: 0.7520 - val_accuracy: 0.8157\n","Epoch 44/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.3385 - accuracy: 0.8893 - val_loss: 0.7112 - val_accuracy: 0.8157\n","Epoch 45/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.3148 - accuracy: 0.8969 - val_loss: 0.7943 - val_accuracy: 0.7888\n","Epoch 46/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.3178 - accuracy: 0.8983 - val_loss: 0.9351 - val_accuracy: 0.7730\n","Epoch 47/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.3454 - accuracy: 0.8848 - val_loss: 0.9103 - val_accuracy: 0.7618\n","Epoch 48/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2943 - accuracy: 0.9030 - val_loss: 0.7127 - val_accuracy: 0.8382\n","Epoch 49/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.3074 - accuracy: 0.8918 - val_loss: 0.6615 - val_accuracy: 0.8292\n","Epoch 50/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.2809 - accuracy: 0.9056 - val_loss: 0.8694 - val_accuracy: 0.7978\n","Epoch 51/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.3323 - accuracy: 0.8907 - val_loss: 0.7120 - val_accuracy: 0.8315\n","Epoch 52/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.2929 - accuracy: 0.9042 - val_loss: 0.7858 - val_accuracy: 0.7955\n","Epoch 53/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.2924 - accuracy: 0.9013 - val_loss: 0.8111 - val_accuracy: 0.8157\n","Epoch 54/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2650 - accuracy: 0.9101 - val_loss: 0.5281 - val_accuracy: 0.8719\n","Epoch 55/200\n","56/56 [==============================] - 7s 123ms/step - loss: 0.2644 - accuracy: 0.9132 - val_loss: 1.5610 - val_accuracy: 0.6966\n","Epoch 56/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.2742 - accuracy: 0.9089 - val_loss: 0.7429 - val_accuracy: 0.8360\n","Epoch 57/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2850 - accuracy: 0.9073 - val_loss: 0.8747 - val_accuracy: 0.8090\n","Epoch 58/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.2562 - accuracy: 0.9137 - val_loss: 0.5942 - val_accuracy: 0.8719\n","Epoch 59/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2295 - accuracy: 0.9216 - val_loss: 0.5809 - val_accuracy: 0.8742\n","Epoch 60/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.2352 - accuracy: 0.9199 - val_loss: 0.7074 - val_accuracy: 0.8449\n","Epoch 61/200\n","56/56 [==============================] - 7s 123ms/step - loss: 0.2423 - accuracy: 0.9140 - val_loss: 0.5100 - val_accuracy: 0.8787\n","Epoch 62/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.2627 - accuracy: 0.9165 - val_loss: 1.0485 - val_accuracy: 0.7663\n","Epoch 63/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2431 - accuracy: 0.9185 - val_loss: 0.6146 - val_accuracy: 0.8449\n","Epoch 64/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2322 - accuracy: 0.9236 - val_loss: 0.7402 - val_accuracy: 0.8180\n","Epoch 65/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.2452 - accuracy: 0.9174 - val_loss: 0.6505 - val_accuracy: 0.8472\n","Epoch 66/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2033 - accuracy: 0.9283 - val_loss: 0.6009 - val_accuracy: 0.8764\n","Epoch 67/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2496 - accuracy: 0.9224 - val_loss: 0.8449 - val_accuracy: 0.8292\n","Epoch 68/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2306 - accuracy: 0.9210 - val_loss: 0.7287 - val_accuracy: 0.8472\n","Epoch 69/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2079 - accuracy: 0.9328 - val_loss: 0.8189 - val_accuracy: 0.8180\n","Epoch 70/200\n","56/56 [==============================] - 7s 120ms/step - loss: 0.2155 - accuracy: 0.9325 - val_loss: 0.7928 - val_accuracy: 0.8494\n","Epoch 71/200\n","56/56 [==============================] - 7s 120ms/step - loss: 0.1828 - accuracy: 0.9359 - val_loss: 0.6065 - val_accuracy: 0.8629\n","Epoch 72/200\n","56/56 [==============================] - 7s 120ms/step - loss: 0.2028 - accuracy: 0.9370 - val_loss: 0.6486 - val_accuracy: 0.8539\n","Epoch 73/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.2085 - accuracy: 0.9297 - val_loss: 1.0229 - val_accuracy: 0.7708\n","Epoch 74/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.2116 - accuracy: 0.9311 - val_loss: 0.6174 - val_accuracy: 0.8764\n","Epoch 75/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.1802 - accuracy: 0.9354 - val_loss: 0.9478 - val_accuracy: 0.7888\n","Epoch 76/200\n","56/56 [==============================] - 7s 123ms/step - loss: 0.1816 - accuracy: 0.9393 - val_loss: 0.8464 - val_accuracy: 0.8270\n","Epoch 77/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.1829 - accuracy: 0.9362 - val_loss: 0.8439 - val_accuracy: 0.8090\n","Epoch 78/200\n","56/56 [==============================] - 7s 120ms/step - loss: 0.2123 - accuracy: 0.9320 - val_loss: 0.7894 - val_accuracy: 0.8382\n","Epoch 79/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.1728 - accuracy: 0.9373 - val_loss: 0.6116 - val_accuracy: 0.8652\n","Epoch 80/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.1803 - accuracy: 0.9376 - val_loss: 0.7313 - val_accuracy: 0.8764\n","Epoch 81/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.1708 - accuracy: 0.9365 - val_loss: 0.7210 - val_accuracy: 0.8337\n","Epoch 82/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.1646 - accuracy: 0.9438 - val_loss: 0.9563 - val_accuracy: 0.8315\n","Epoch 83/200\n","56/56 [==============================] - 7s 120ms/step - loss: 0.1749 - accuracy: 0.9379 - val_loss: 0.7291 - val_accuracy: 0.8697\n","Epoch 84/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.1978 - accuracy: 0.9359 - val_loss: 0.8056 - val_accuracy: 0.8539\n","Epoch 85/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.1866 - accuracy: 0.9373 - val_loss: 0.8510 - val_accuracy: 0.8315\n","Epoch 86/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.1718 - accuracy: 0.9413 - val_loss: 0.6204 - val_accuracy: 0.8787\n","Epoch 87/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.1591 - accuracy: 0.9505 - val_loss: 0.6059 - val_accuracy: 0.8787\n","Epoch 88/200\n","56/56 [==============================] - 7s 121ms/step - loss: 0.1719 - accuracy: 0.9446 - val_loss: 0.8508 - val_accuracy: 0.8270\n","Epoch 89/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.1638 - accuracy: 0.9474 - val_loss: 0.7714 - val_accuracy: 0.8270\n","Epoch 90/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.1666 - accuracy: 0.9472 - val_loss: 0.9470 - val_accuracy: 0.8315\n","Epoch 91/200\n","56/56 [==============================] - 7s 122ms/step - loss: 0.1499 - accuracy: 0.9466 - val_loss: 0.7792 - val_accuracy: 0.8607\n"]}],"source":["# Callbacks and train model\n","\n","callbacks = [\n","    keras.callbacks.ModelCheckpoint(\n","        filepath=\"saved_models/model_tuned.keras\",\n","        save_best_only=True,\n","        monitor=\"val_loss\"\n","    ),\n","    keras.callbacks.EarlyStopping(\n","        patience=30,\n","        monitor='val_loss'\n","    )\n","]\n","\n","history = model_best_hps.fit(\n","                train_dataset,\n","                epochs=200,\n","                batch_size=64,\n","                validation_data=validation_dataset,\n","                callbacks=callbacks\n",")"]},{"cell_type":"code","execution_count":null,"id":"e3f66a30-043b-4f54-b599-df73155fe18b","metadata":{"id":"e3f66a30-043b-4f54-b599-df73155fe18b","tags":[]},"outputs":[],"source":["# Visualization\n","hist_df = pd.DataFrame(history.history)\n","loss = px.scatter(hist_df['loss'])\n","val_loss = px.line(hist_df['val_loss'])\n","accuracy = px.scatter(hist_df['accuracy'])\n","val_accuracy = px.line(hist_df['val_accuracy'])\n","\n","fig = make_subplots(cols=2, rows=1, subplot_titles=(\"Loss\", \"Accuracy\"))\n","fig.add_trace(loss.data[0], col=1, row=1)\n","fig.add_trace(val_loss.data[0], col=1, row=1)\n","fig.add_trace(accuracy.data[0], col=2, row=1)\n","fig.add_trace(val_accuracy.data[0], col=2, row=1)\n","fig.update_layout(height=600)\n","\n","fig.show()"]},{"cell_type":"markdown","source":["![Accuracy and loss](https://www.dropbox.com/s/j0obaviyp8si4pb/hypersearch.png?raw=1)"],"metadata":{"id":"of4zl7spuliQ"},"id":"of4zl7spuliQ"},{"cell_type":"markdown","id":"f4c1de33-d6e5-4aea-97a8-86cc3b7f2409","metadata":{"id":"f4c1de33-d6e5-4aea-97a8-86cc3b7f2409"},"source":["We compare against C and E because C performed well having a simple architecture and E because it performed better."]},{"cell_type":"code","execution_count":null,"id":"682f2e03-d14e-41d8-8a31-e71d72c5d325","metadata":{"id":"682f2e03-d14e-41d8-8a31-e71d72c5d325","outputId":"78ee677c-4275-4f57-eccb-0bd29006cf09","tags":[]},"outputs":[{"name":"stdout","output_type":"stream","text":["7/7 [==============================] - 0s 17ms/step - loss: 0.5692 - accuracy: 0.8260\n","7/7 [==============================] - 0s 61ms/step - loss: 0.4946 - accuracy: 0.8627\n","7/7 [==============================] - 0s 25ms/step - loss: 0.5723 - accuracy: 0.8529\n","Model C: 82.60% of accuracy\n","Model E: 86.27% of accuracy\n","Model tuned: 85.29% of accuracy\n"]}],"source":["# Load models\n","\n","model_C = load_model(\"saved_models/model_handcrafted_C.keras\")\n","model_D = load_model(\"saved_models/model_handcrafted_D.keras\")\n","model_tuned = load_model(\"saved_models/model_tuned.keras\")\n","\n","\n","_, model_C_acc = model_C.evaluate(test_dataset)\n","_, model_D_acc = model_E.evaluate(test_dataset)\n","_, model_tuned_acc = model_tuned.evaluate(test_dataset)\n","\n","print(\n","    f\"Model C: {model_C_acc * 100:.2f}% of accuracy\\n\"\n","    f\"Model E: {model_D_acc * 100:.2f}% of accuracy\\n\"\n","    f\"Model tuned: {model_tuned_acc * 100:.2f}% of accuracy\"\n","      )"]},{"cell_type":"markdown","id":"n9br7xwPIMM6","metadata":{"id":"n9br7xwPIMM6"},"source":["Model tuned, while having the same performance, is less complex and trains faster than D, that makes it a better model."]}],"metadata":{"colab":{"provenance":[]},"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.10"}},"nbformat":4,"nbformat_minor":5}