{"id":22039516,"url":"https://github.com/mpolinowski/keras_transfer_learning_2023","last_synced_at":"2026-04-13T21:03:29.433Z","repository":{"id":234831423,"uuid":"617354340","full_name":"mpolinowski/keras_transfer_learning_2023","owner":"mpolinowski","description":"Using Keras models and datasets to build custom prediction models","archived":false,"fork":false,"pushed_at":"2023-03-22T08:10:52.000Z","size":13348,"stargazers_count":0,"open_issues_count":0,"forks_count":0,"subscribers_count":1,"default_branch":"master","last_synced_at":"2025-01-28T19:17:43.006Z","etag":null,"topics":["food-101","keras-application","tensorflow-datasets","transfer-learning"],"latest_commit_sha":null,"homepage":"","language":"Jupyter Notebook","has_issues":true,"has_wiki":null,"has_pages":null,"mirror_url":null,"source_name":null,"license":null,"status":null,"scm":"git","pull_requests_enabled":true,"icon_url":"https://github.com/mpolinowski.png","metadata":{"files":{"readme":"README.md","changelog":null,"contributing":null,"funding":null,"license":null,"code_of_conduct":null,"threat_model":null,"audit":null,"citation":null,"codeowners":null,"security":null,"support":null,"governance":null,"roadmap":null,"authors":null,"dei":null,"publiccode":null,"codemeta":null}},"created_at":"2023-03-22T08:07:14.000Z","updated_at":"2023-04-02T09:31:46.000Z","dependencies_parsed_at":null,"dependency_job_id":"ac786d8a-b940-445e-bf35-b6b9812a2f48","html_url":"https://github.com/mpolinowski/keras_transfer_learning_2023","commit_stats":null,"previous_names":["mpolinowski/keras_transfer_learning_2023"],"tags_count":0,"template":false,"template_full_name":null,"repository_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpolinowski%2Fkeras_transfer_learning_2023","tags_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpolinowski%2Fkeras_transfer_learning_2023/tags","releases_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpolinowski%2Fkeras_transfer_learning_2023/releases","manifests_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories/mpolinowski%2Fkeras_transfer_learning_2023/manifests","owner_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners/mpolinowski","download_url":"https://codeload.github.com/mpolinowski/keras_transfer_learning_2023/tar.gz/refs/heads/master","host":{"name":"GitHub","url":"https://github.com","kind":"github","repositories_count":245104529,"owners_count":20561380,"icon_url":"https://github.com/github.png","version":null,"created_at":"2022-05-30T11:31:42.601Z","updated_at":"2022-07-04T15:15:14.044Z","host_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub","repositories_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repositories","repository_names_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/repository_names","owners_url":"https://repos.ecosyste.ms/api/v1/hosts/GitHub/owners"}},"keywords":["food-101","keras-application","tensorflow-datasets","transfer-learning"],"created_at":"2024-11-30T11:11:05.446Z","updated_at":"2026-04-13T21:03:29.392Z","avatar_url":"https://github.com/mpolinowski.png","language":"Jupyter Notebook","funding_links":[],"categories":[],"sub_categories":[],"readme":"---\njupyter:\n  jupytext:\n    formats: ipynb,md\n    text_representation:\n      extension: .md\n      format_name: markdown\n      format_version: '1.3'\n      jupytext_version: 1.14.4\n  kernelspec:\n    display_name: Python 3 (ipykernel)\n    language: python\n    name: python3\n---\n\n# Keras Applications \u0026 Tensorflow Datasets\n\nUsing the EfficientNetB0 model from Keras applications with the Food-101 dataset from [Tensorflow datasets](https://tensorflow.google.cn/datasets/overview) to build a food image classifier prediction API.\n\n```python\nimport datetime\nimport matplotlib.image as mpimg\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport os\nimport pandas as pd\nimport random\nfrom sklearn.metrics import accuracy_score, classification_report\nimport tensorflow as tf\nfrom tensorflow.keras import layers, mixed_precision\nimport tensorflow_datasets as tfds\n```\n\n```python\n# import helper functions from helper.py\nfrom helper import (create_tensorboard_callback,\n                    create_checkpoint_callback,\n                    create_early_stop_callback,\n                    create_reduce_learning_rate_callback,\n                    plot_accuracy_curves,\n                    combine_training_curves,\n                    data_augmentation_layer_no_rescaling,\n                    plot_confusion_matrix)\n```\n\n```python\n# global variables\nSEED = 42\nBATCH_SIZE = 32\nIMG_DIM = 224\n```\n\n## Tensorflow Datasets\n\n\u003e `pip install tensorflow-datasets`\n\n[TFDS Overview](https://www.tensorflow.org/datasets/catalog/overview)\n\n\n```python\n# list all available datasets\navailable_datasets = tfds.list_builders()\navailable_datasets\n```\n\n### Downloading Datasets\n\n```python\n# download the food-101 dataset (download size: 4.65 GiB)\n# https://www.tensorflow.org/datasets/catalog/food101 \n\n(train_data, test_data), ds_info = tfds.load(name=\"food101\",\n                                           split=[\"train\", \"validation\"],\n                                           shuffle_files=True,\n                                           as_supervised=True,\n                                           with_info=True)\n\n# Dataset food101 downloaded and prepared to /home/myuser/tensorflow_datasets/food101/2.0.0.\n# Subsequent calls will reuse this data.\n```\n\n### Exploring the Dataset\n\n```python\n# get features\nprint(ds_info.features)\n\n# FeaturesDict({\n#     'image': Image(shape=(None, None, 3), dtype=uint8),\n#     'label': ClassLabel(shape=(), dtype=int64, num_classes=101),\n# })\n\n# get classnames\nclass_names = ds_info.features[\"label\"].names\nprint(class_names[:5])\n# ['apple_pie', 'baby_back_ribs', 'baklava', 'beef_carpaccio', 'beef_tartare']\n```\n\n```python\n# explore trainings data\nprint(train_data)\n# \u003cPrefetchDataset element_spec=(TensorSpec(shape=(None, None, 3), dtype=tf.uint8, name=None), TensorSpec(shape=(), dtype=tf.int64, name=None))\u003e\n\n# get one sample image\ntrain_one_sample = train_data.take(1)\nprint(train_one_sample)\n# \u003cTakeDataset element_spec=(TensorSpec(shape=(None, None, 3), dtype=tf.uint8, name=None), TensorSpec(shape=(), dtype=tf.int64, name=None))\u003e\n\nfor image, label in train_one_sample:\n    print(f\"\"\"\n        Image Shape: {image.shape}\n        Image Datatype: {image.dtype}\n        Class Tensor: {label}\n        Classname: {class_names[label.numpy()]}\n    \"\"\")\n\n# images are:\n# * 3 colour channels but not standardized to 224x224\n# * datatype uint8 needs to be changed (float16/float32)\n# * classes are not 1-hot-encoded =\u003e sparse-categorical-crossentropy loss-function needed\n\n#     Image Shape: (384, 512, 3)\n#     Image Datatype: \u003cdtype: 'uint8'\u003e\n#     Class Tensor: 70\n#     Classname: pad_thai\n```\n\n```python\n# what do the images look like?\nimage\n\n# \u003ctf.Tensor: shape=(384, 512, 3), dtype=uint8, numpy=\n# array([[[230, 229, 183],\n#         [231, 230, 184],\n#         [232, 231, 183],\n\n#        ...,\n       \n#         [243, 245, 224],\n#         [244, 246, 225],\n#         [245, 247, 226]]], dtype=uint8)\u003e\n\n## min/max values\ntf.reduce_min(image), tf.reduce_max(image)\n\n# colour values range from 0 - 255 -\u003e needs to be normalized\n# (\u003ctf.Tensor: shape=(), dtype=uint8, numpy=0\u003e,\n#  \u003ctf.Tensor: shape=(), dtype=uint8, numpy=255\u003e)\n```\n\n```python\n# plot the image\nplt.imshow(image.numpy())\nplt.title(class_names[label.numpy()]+ \" \" + str(image.shape))\nplt.axis('off')\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_21.png)\n\n\n### Preprocess Dataset\n\n* Reshape image `(x, y, 3)` =\u003e `(224, 224, 3)`\n* Convert dType `unit8` =\u003e `float32`\n* Normalize (_not necessary for EfficientNet models_)\n\n```python\ndef preprocess_image(image, label, image_shape=224):\n    image = tf.image.resize(image, [image_shape, image_shape])\n    # normalization not needed for efficientnet\n    # image = image/255\n    return tf.cast(image, tf.float32), label\n```\n\n```python\n# test preprocessing function\npreprocessed_image = preprocess_image(image, class_names[label.numpy()], image_shape=IMG_DIM)\npreprocessed_image\n\n# (\u003ctf.Tensor: shape=(224, 224, 3), dtype=float32, numpy=\n#  array([[[229.46939 , 228.46939 , 181.7551  ],\n#          [229.59184 , 228.94897 , 180.2347  ],\n#          [224.14796 , 224.14796 , 171.71939 ],\n \n#         ...,\n        \n#          [241.79082 , 243.79082 , 222.79082 ],\n#          [241.66327 , 243.66327 , 222.66327 ],\n#          [242.80103 , 244.80103 , 223.80103 ]]], dtype=float32)\u003e,\n#  'pad_thai')\n```\n\n\u003c!-- #region --\u003e\n#### Prepare Batched Training/Testing Pipeline\n\n\u003e [Data Performance](https://www.tensorflow.org/guide/data_performance)\n\n\n* map preprocess function to dataset\n* set parallel calls to autotune = use all available threads\n* shuffle and batch dataset and set prefetch buffer to autotune\n\u003c!-- #endregion --\u003e\n\n```python\ntraining_dataset = train_data.map(map_func=preprocess_image, num_parallel_calls=tf.data.AUTOTUNE)\ntraining_dataset = training_dataset.shuffle(buffer_size=1000).batch(batch_size=BATCH_SIZE).prefetch(buffer_size=tf.data.AUTOTUNE)   \n\ntesting_dataset = test_data.map(map_func=preprocess_image, num_parallel_calls=tf.data.AUTOTUNE)\ntesting_dataset = testing_dataset.batch(BATCH_SIZE).prefetch(tf.data.AUTOTUNE)\n\ntraining_dataset, testing_dataset\n# (\u003cPrefetchDataset element_spec=(TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32, name=None), TensorSpec(shape=(None,), dtype=tf.int64, name=None))\u003e,\n#  \u003cPrefetchDataset element_spec=(TensorSpec(shape=(None, 224, 224, 3), dtype=tf.float32, name=None), TensorSpec(shape=(None,), dtype=tf.int64, name=None))\u003e)\n```\n\n### Tensorflow Callbacks\n\n```python\n# import callbacks from helper.py\n\n## INITIAL TEST\n### TensorBoard\ntensorboard_dir = '../tensorboard/food101'\nexperiment_name = 'efficientnetb0_food101_full'\n\ntensorboard_callback = create_tensorboard_callback(tensorboard_dir, experiment_name)\n\n### Checkpoints\ncheckpoint_dir = '../checkpoints/food101'\n\ncheckpoint_callback = create_checkpoint_callback(checkpoint_dir, experiment_name)\n\n                                          \n### Early Stop\nearly_stop_callback = create_early_stop_callback(monitor='val_loss',\n                              min_delta=0.0001,\n                              patience=10,\n                              restore_best_weights=True)\n\n## AUGMENTED TEST\n### TensorBoard\nexperiment_name_augmented = 'efficientnetb0_food101_augmented_full'\n\ntensorboard_augmented_callback = create_tensorboard_callback(tensorboard_dir, experiment_name_augmented)\n\n### Checkpoints\ncheckpoint_augmented_callback = create_checkpoint_callback(checkpoint_dir, experiment_name_augmented)\n\n                                          \n### Reduce Learning Rate\nreduce_learning_rate_callback = create_reduce_learning_rate_callback(monitor=\"val_loss\",  \n                                        factor=0.2,\n                                        patience=2,\n                                        min_lr=1e-7)\n\n## V2 MODEL M TEST\n### TensorBoard\nexperiment_name_v2 = 'efficientnetb0_food101_v2_full'\n\ntensorboard_v2_callback = create_tensorboard_callback(tensorboard_dir, experiment_name_v2)\n\n### Checkpoints\ncheckpoint_v2_callback = create_checkpoint_callback(checkpoint_dir, experiment_name_v2)  \n\n\n## V2 MODEL B0 TEST\n### TensorBoard\nexperiment_name_v2_b0 = 'efficientnetb0_food101_v2_b0_full'\n\ntensorboard_v2_b0_callback = create_tensorboard_callback(tensorboard_dir, experiment_name_v2)\n\n### Checkpoints\ncheckpoint_v2_b0_callback = create_checkpoint_callback(checkpoint_dir, experiment_name_v2)                                            \n```\n\n## Mixed Precision Training\n\n```python\n# configure mixed precision training\nmixed_precision.set_global_policy(\"mixed_float16\")\nmixed_precision.global_policy()\n\n# \u003cPolicy \"mixed_float16\"\u003e\n\n# WARNING:tensorflow:Mixed precision compatibility check (mixed_float16): WARNING\n# Your GPU may run slowly with dtype policy mixed_float16 because it does not have compute capability of at least 7.0. Your GPU:\n#   NVIDIA GeForce GTX 1060 6GB, compute capability 6.1\n# See https://developer.nvidia.com/cuda-gpus for a list of GPUs and their compute capabilities.\n```\n\n```python\n# load the base model\nbase_model = tf.keras.applications.EfficientNetB0(include_top=False)\nbase_model.trainable = False\n```\n\n```python\n# create functional model\ninputs = layers.Input(shape=(IMG_DIM, IMG_DIM, 3), name=\"input_layer\")\n# normalization not needed for EfficientNet models\n# x = layers.Rescaling(1./255)(x)\n# freeze base layers in inference mode\nx = base_model(inputs, training=False)\nx = layers.GlobalAveragePooling2D()(x)\nx = layers.Dense(len(class_names))(x)\n# mixed precision requires outputlayer to be of dtype = tf.float32\n# above mixed mixed float16 / below float32 separation\noutputs = layers.Activation(\"softmax\", dtype=tf.float32, name=\"softmax_output_float32\")(x)\nmodel = tf.keras.Model(inputs, outputs)\n\n# labels are not 1-hot encoded =\u003e sparse_categorical instead of categorical\nmodel.compile(loss=\"sparse_categorical_crossentropy\",\n             optimizer= tf.keras.optimizers.Adam(learning_rate=1e-3),\n             metrics=[\"accuracy\"])\n```\n\n```python\nprint(model.summary())\n\n# Model: \"model_1\"\n# _________________________________________________________________\n#  Layer (type)                Output Shape              Param #   \n# =================================================================\n#  input_layer (InputLayer)    [(None, 224, 224, 3)]     0\n#  efficientnetb0 (Functional)  (None, None, None, 1280)  4049571\n#  global_average_pooling2d_3   (None, 1280)             0         \n#  (GlobalAveragePooling2D)\n#  dense_3 (Dense)             (None, 101)               129381\n#  softmax_output_float32 (Act  (None, 101)              0         \n#  ivation)\n# =================================================================\n# Total params: 4,178,952\n# Trainable params: 129,381\n# Non-trainable params: 4,049,571\n# _________________________________________________________________\n\nfor layer in model.layers:\n    print(layer.name, layer.trainable, layer.dtype, layer.dtype_policy)\n    \n# input_layer True float32 \u003cPolicy \"float32\"\u003e\n# efficientnetb0 False float32 \u003cPolicy \"mixed_float16\"\u003e\n# global_average_pooling2d_3 True float32 \u003cPolicy \"mixed_float16\"\u003e\n# dense_3 True float32 \u003cPolicy \"mixed_float16\"\u003e\n# softmax_output_float32 True float32 \u003cPolicy \"float32\"\u003e\n```\n\n### Training Run\n\n```python\ntf.random.set_seed(SEED)\nfeature_extraction_epochs = 5\n\nfeature_extraction_history = model.fit(training_dataset,\n                                         epochs=feature_extraction_epochs,\n                                         steps_per_epoch=len(training_dataset),\n                                         validation_data=testing_dataset,\n                                         # evaluate performance on 15% of the testing dataset\n                                         validation_steps=int(0.15 * len(testing_dataset)),\n                                         callbacks=[tensorboard_callback,\n                                                    checkpoint_callback,\n                                                    early_stop_callback])\n\n# Epoch 1/5\n# 221s 89ms/step - loss: 1.7176 - accuracy: 0.5821 - val_loss: 1.1405 - val_accuracy: 0.6891\n# Epoch 2/5\n# 206s 87ms/step - loss: 1.1998 - accuracy: 0.6892 - val_loss: 1.0268 - val_accuracy: 0.7156\n# Epoch 3/5\n# 215s 91ms/step - loss: 1.0541 - accuracy: 0.7243 - val_loss: 0.9917 - val_accuracy: 0.7317\n# Epoch 4/5\n# 216s 91ms/step - loss: 0.9598 - accuracy: 0.7468 - val_loss: 0.9809 - val_accuracy: 0.7296\n# Epoch 5/5\n# 213s 90ms/step - loss: 0.8884 - accuracy: 0.7657 - val_loss: 0.9658 - val_accuracy: 0.7378\n```\n\n```python\nfeature_extraction_results = model.evaluate(testing_dataset)\nprint(feature_extraction_results)\n\n# [0.9763504266738892, 0.7352079153060913]\n```\n\n### Model Fine-tuning\n\n```python\n# unfreeze entire model\nbase_model.trainable = True\n\n# keep only the last 5 layers trainable\nfor layer in base_model.layers[:-5]:\n    layer.trainable = False\n```\n\n```python\n# recompile the model with the new basemodel\n### to prevent overfitting / to better hold on to pre-training\n### the learning rate during fine-tuning should be lowered 10x\n### default Adam(lr)=1e-3 =\u003e 1e-4\nmodel.compile(loss='sparse_categorical_crossentropy',\n               optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),\n               metrics=['accuracy'])\n```\n\n```python\n# continue training\ntf.random.set_seed(SEED)\nfine_tuning_epochs = feature_extraction_epochs + 5\n\nfine_tuning_history = model.fit(\n                            training_dataset,\n                            epochs=fine_tuning_epochs,\n                            # start from last pre-training checkpoint\n                            # training from epoch 6 - 10\n                            initial_epoch = feature_extraction_history.epoch[-1],\n                            steps_per_epoch=len(training_dataset),\n                            validation_data=testing_dataset,\n                            # evaluate performance on 15% of the testing dataset\n                            validation_steps=int(0.15 * len(testing_dataset)),\n                            callbacks=[tensorboard_callback,\n                                       checkpoint_callback])\n\n# Epoch 5/10\n# 227s 93ms/step - loss: 0.7658 - accuracy: 0.7958 - val_loss: 0.9151 - val_accuracy: 0.7476\n# Epoch 6/10\n# 226s 95ms/step - loss: 0.6486 - accuracy: 0.8283 - val_loss: 0.9096 - val_accuracy: 0.7476\n# Epoch 7/10\n# 225s 95ms/step - loss: 0.5643 - accuracy: 0.8527 - val_loss: 0.9090 - val_accuracy: 0.7516\n# Epoch 8/10\n# 225s 95ms/step - loss: 0.4946 - accuracy: 0.8738 - val_loss: 0.9063 - val_accuracy: 0.7564\n# Epoch 9/10\n# 231s 97ms/step - loss: 0.4352 - accuracy: 0.8914 - val_loss: 0.9164 - val_accuracy: 0.7585\n# Epoch 10/10\n# 233s 98ms/step - loss: 0.3821 - accuracy: 0.9061 - val_loss: 0.9256 - val_accuracy: 0.7569\n\n```\n\n```python\n# evaluate performance on whole dataset\nfine_tuning_results = model.evaluate(testing_dataset)\nprint(fine_tuning_results)\n\n# Feature Extraction\n# [0.9763504266738892, 0.7352079153060913]\n# Fine-Tuning\n# [0.9353731274604797, 0.7551287412643433]\n```\n\n### Model Evaluation\n\n#### Accuracy and Loss\n\n```python\n# print accuracy curves\nplot_accuracy_curves(feature_extraction_history, \"Feature Extraction\", fine_tuning_history, \"Fine-Tuning\")\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_22.png)\n\n```python\n# the validation accuracy increase keeps slowing while training\n# accuracy goes up this points to an overfitting problem\ncombine_training_curves(feature_extraction_history, fine_tuning_history, pretraining_epochs=5)\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_23.png)\n\n\n#### Predictions\n\n```python\ny_pred = []  # store predicted labels\ny_true = []  # store true labels\n\n# iterate over the dataset\nfor image_batch, label_batch in testing_dataset:   # use dataset.unbatch() with repeat\n   # append true labels\n   y_true.append(label_batch)\n   # compute predictions\n   preds = model.predict(image_batch)\n   # append predicted labels\n   y_pred.append(np.argmax(preds, axis = - 1))\n\n# convert the true and predicted labels into tensors\ncorrect_labels = tf.concat([item for item in y_true], axis = 0)\npredicted_labels = tf.concat([item for item in y_pred], axis = 0)\n\ncorrect_labels, predicted_labels\n\n# (\u003ctf.Tensor: shape=(25250,), dtype=int64, numpy=array([37, 99, 40, ..., 56, 46, 89])\u003e,\n#  \u003ctf.Tensor: shape=(25250,), dtype=int64, numpy=array([37, 36, 40, ..., 11, 46, 89])\u003e)\n```\n\n```python\nplot_confusion_matrix(y_pred=predicted_labels,\n                      y_true=correct_labels,\n                      classes=class_names,\n                      figsize = (88, 88),\n                      text_size=8)\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_24.png)\n\n```python\n# Load TensorBoard\n%load_ext tensorboard\n%tensorboard --logdir '../tensorboard/food101/'\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_25.png)\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_26.png)\n\n\n## Input Data Augmentation\n\nUsing Tensorflow image augmentation function to \"virtually\" diversify the dataset and tackling the overfitting issue.\n\n```python\n# helper function to create an augmented model\ndata_augmentation_layer_no_rescaling = tf.keras.Sequential([\n    tf.keras.layers.RandomFlip(\"horizontal_and_vertical\"),\n    tf.keras.layers.RandomRotation(0.2),\n    tf.keras.layers.RandomZoom(0.2),\n    tf.keras.layers.RandomTranslation(\n            height_factor=(-0.2, 0.3),\n            width_factor=(-0.2, 0.3),\n            fill_mode='reflect',\n            interpolation='bilinear'),\n    tf.keras.layers.RandomContrast(0.2),\n    tf.keras.layers.RandomBrightness(0.2)\n], name=\"data_augmentation\")\n```\n\n```python\n# load the base model a second time\nbase_model_augmented = tf.keras.applications.EfficientNetB0(include_top=False)\nbase_model_augmented.trainable = False\n```\n\n```python\n# create augmented functional model\ninputs = layers.Input(shape=(IMG_DIM, IMG_DIM, 3), name=\"input_layer\")\nx = data_augmentation_layer_no_rescaling(inputs)\n# normalization not needed for EfficientNet models\n# x = layers.Rescaling(1./255)(x)\n# freeze base layers in inference mode\nx = base_model_augmented(x, training=False)\nx = layers.GlobalAveragePooling2D()(x)\nx = layers.Dense(len(class_names))(x)\n# mixed precision requires outputlayer to be of dtype = tf.float32\n# above mixed mixed float16 / below float32 separation\noutputs = layers.Activation(\"softmax\", dtype=tf.float32, name=\"softmax_output_float32\")(x)\nmodel2_augmented = tf.keras.Model(inputs, outputs)\n\n# labels are not 1-hot encoded =\u003e sparse_categorical instead of categorical\nmodel2_augmented.compile(loss=\"sparse_categorical_crossentropy\",\n             optimizer= tf.keras.optimizers.Adam(learning_rate=1e-3),\n             metrics=[\"accuracy\"])\n```\n\n```python\ntf.random.set_seed(SEED)\nfeature_extraction_augmented_epochs = 5\n\nfeature_extraction_augmented_history = model2_augmented.fit(training_dataset,\n                                             epochs=feature_extraction_epochs,\n                                             steps_per_epoch=len(training_dataset),\n                                             validation_data=testing_dataset,\n                                             # evaluate performance on 15% of the testing dataset\n                                             validation_steps=int(0.15 * len(testing_dataset)),\n                                             callbacks=[tensorboard_augmented_callback,\n                                                        checkpoint_augmented_callback,\n                                                        reduce_learning_rate_callback,\n                                                        early_stop_callback])\n\n# Epoch 1/5\n# 2368/2368 [==============================] - 647s 269ms/step - loss: 2.5783 - accuracy: 0.3880 - val_loss: 1.6074 - val_accuracy: 0.5792 - lr: 0.0010\n# Epoch 2/5\n# 591s 249ms/step - loss: 2.0982 - accuracy: 0.4789 - val_loss: 1.4961 - val_accuracy: 0.5927 - lr: 0.0010\n# Epoch 3/5\n# 581s 245ms/step - loss: 1.9760 - accuracy: 0.5060 - val_loss: 1.4502 - val_accuracy: 0.6001 - lr: 0.0010\n# Epoch 4/5\n# 580s 245ms/step - loss: 1.9136 - accuracy: 0.5202 - val_loss: 1.4311 - val_accuracy: 0.6147 - lr: 0.0010\n# Epoch 5/5\n# 594s 251ms/step - loss: 1.8632 - accuracy: 0.5288 - val_loss: 1.4012 - val_accuracy: 0.6160 - lr: 0.0010\n```\n\n```python\n# unfreeze entire model\nbase_model_augmented.trainable = True\n\n# keep only the last 5 layers trainable\nfor layer in base_model_augmented.layers[:-5]:\n    layer.trainable = False\n```\n\n```python\n# recompile the model with the new basemodel\nmodel2_augmented.compile(loss='sparse_categorical_crossentropy',\n               optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),\n               metrics=['accuracy'])\n```\n\n```python\n# continue training\ntf.random.set_seed(SEED)\nfine_tuning_augmented_epochs = feature_extraction_augmented_epochs + 5\n\nfine_tuning_augmented_history = model2_augmented.fit(\n                            training_dataset,\n                            epochs=fine_tuning_augmented_epochs,\n                            # start from last pre-training checkpoint\n                            # training from epoch 6 - 10\n                            initial_epoch = feature_extraction_augmented_history.epoch[-1],\n                            steps_per_epoch=len(training_dataset),\n                            validation_data=testing_dataset,\n                            # evaluate performance on 15% of the testing dataset\n                            validation_steps=int(0.15 * len(testing_dataset)),\n                            callbacks=[tensorboard_augmented_callback,\n                                       checkpoint_augmented_callback,\n                                       reduce_learning_rate_callback,\n                                       early_stop_callback])\n\n# Epoch 5/10\n# 2368/2368 [==============================] - 699s 290ms/step - loss: 1.6879 - accuracy: 0.5703 - val_loss: 1.3540 - val_accuracy: 0.6316 - lr: 1.0000e-04\n# Epoch 6/10\n# 661s 279ms/step - loss: 1.5625 - accuracy: 0.5996 - val_loss: 1.3304 - val_accuracy: 0.6380 - lr: 1.0000e-04\n# Epoch 7/10\n# 712s 300ms/step - loss: 1.4884 - accuracy: 0.6163 - val_loss: 1.3158 - val_accuracy: 0.6414 - lr: 1.0000e-04\n# Epoch 8/10\n# 710s 299ms/step - loss: 1.4279 - accuracy: 0.6287 - val_loss: 1.2938 - val_accuracy: 0.6486 - lr: 1.0000e-04\n# Epoch 9/10\n# 713s 300ms/step - loss: 1.3782 - accuracy: 0.6402 - val_loss: 1.2849 - val_accuracy: 0.6499 - lr: 1.0000e-04\n# Epoch 10/10\n# 714s 301ms/step - loss: 1.3448 - accuracy: 0.6497 - val_loss: 1.2670 - val_accuracy: 0.6510 - lr: 1.0000e-04\n\n```\n\n### Model Evaluation\n\n#### Accuracy and Loss\n\n```python\n# the validation accuracy increase keeps slowing while training\ncombine_training_curves(feature_extraction_augmented_history, fine_tuning_augmented_history, pretraining_epochs=5)\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_27.png)\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/nice.gif)\n\n\u003c!-- #region --\u003e\n## Base Model Complexity\n\nEven though the model above lead to a worse accuracy over 5+5 epochs it seems to have solved the overfitting issue. Both the training and validation metrics stick together and keep on improving slowly. We can keep this model running for longer time and the accuracy will keep on rising.\n\n\nBut before running this on a night shift I want to first try out a slightly complexer version of the EfficientNet model. I also noticed that I used [version 1](https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet/EfficientNetB0) instead of [version 2](https://www.tensorflow.org/api_docs/python/tf/keras/applications/efficientnet_v2/EfficientNetV2B0).\n\n\nVersion 2 of EfficientNet offers a small, medium and large model. According to [this paper](https://arxiv.org/abs/2104.00298) the medium model offers a much higher accuracy without a significant increase in parameters:\n\n\n![EfficientNetV2: Smaller Models and Faster Training](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_28.png)\n\u003c!-- #endregion --\u003e\n\n```python\n# load the version 2 base model\nbase_model_v2 = tf.keras.applications.efficientnet_v2.EfficientNetV2M(include_top=False)\nbase_model_v2.trainable = False\n```\n\n```python\n# create augmented functional model\ninputs = layers.Input(shape=(IMG_DIM, IMG_DIM, 3), name=\"input_layer\")\nx = data_augmentation_layer_no_rescaling(inputs)\n# normalization not needed for EfficientNet models\n# x = layers.Rescaling(1./255)(x)\n# freeze base layers in inference mode\nx = base_model_v2(x, training=False)\nx = layers.GlobalAveragePooling2D()(x)\nx = layers.Dense(len(class_names))(x)\n# mixed precision requires outputlayer to be of dtype = tf.float32\n# above mixed mixed float16 / below float32 separation\noutputs = layers.Activation(\"softmax\", dtype=tf.float32, name=\"softmax_output_float32\")(x)\nmodel_v2 = tf.keras.Model(inputs, outputs)\n\n# labels are not 1-hot encoded =\u003e sparse_categorical instead of categorical\nmodel_v2.compile(loss=\"sparse_categorical_crossentropy\",\n             optimizer= tf.keras.optimizers.Adam(learning_rate=1e-3),\n             metrics=[\"accuracy\"])\n```\n\n```python\ntf.random.set_seed(SEED)\nfeature_extraction_v2_epochs = 5\n\nfeature_extraction_v2_history = model_v2.fit(training_dataset,\n                                             epochs=feature_extraction_v2_epochs,\n                                             steps_per_epoch=len(training_dataset),\n                                             validation_data=testing_dataset,\n                                             # evaluate performance on 15% of the testing dataset\n                                             validation_steps=int(0.15 * len(testing_dataset)),\n                                             callbacks=[tensorboard_v2_callback,\n                                                        checkpoint_v2_callback,\n                                                        reduce_learning_rate_callback,\n                                                        early_stop_callback])\n\n\n# Epoch 1/5\n# 1205s 498ms/step - loss: 2.7086 - accuracy: 0.3647 - val_loss: 1.6947 - val_accuracy: 0.5535 - lr: 0.0010\n# Epoch 2/5\n# 1166s 492ms/step - loss: 2.2371 - accuracy: 0.4511 - val_loss: 1.5539 - val_accuracy: 0.5842 - lr: 0.0010\n# Epoch 3/5\n# 1168s 493ms/step - loss: 2.1267 - accuracy: 0.4738 - val_loss: 1.4823 - val_accuracy: 0.6030 - lr: 0.0010\n# Epoch 4/5\n# 1166s 492ms/step - loss: 2.0640 - accuracy: 0.4877 - val_loss: 1.4556 - val_accuracy: 0.6102 - lr: 0.0010\n# Epoch 5/5\n# 1156s 488ms/step - loss: 2.0266 - accuracy: 0.4951 - val_loss: 1.4178 - val_accuracy: 0.6147 - lr: 0.0010\n```\n\n```python\n# unfreeze entire model\nbase_model_v2.trainable = True\n\n# keep only the last 5 layers trainable\nfor layer in base_model_v2.layers[:-5]:\n    layer.trainable = False\n```\n\n```python\n# recompile the model with the new basemodel\nmodel_v2.compile(loss='sparse_categorical_crossentropy',\n               optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),\n               metrics=['accuracy'])\n```\n\n```python\n# continue training\ntf.random.set_seed(SEED)\nfine_tuning_v2_epochs = feature_extraction_v2_epochs + 25\n\nfine_tuning_v2_history = model_v2.fit(\n                            training_dataset,\n                            epochs=fine_tuning_v2_epochs,\n                            # start from last pre-training checkpoint\n                            # training from epoch 6 - 10\n                            initial_epoch = feature_extraction_v2_history.epoch[-1],\n                            steps_per_epoch=len(training_dataset),\n                            validation_data=testing_dataset,\n                            # evaluate performance on 15% of the testing dataset\n                            validation_steps=int(0.15 * len(testing_dataset)),\n                            callbacks=[tensorboard_v2_callback,\n                                       checkpoint_v2_callback,\n                                       reduce_learning_rate_callback,\n                                       early_stop_callback])\n\n# Epoch 5/10\n# 1150s 478ms/step - loss: 2.1729 - accuracy: 0.4661 - val_loss: 1.3997 - val_accuracy: 0.6247 - lr: 1.0000e-04\n# Epoch 6/30\n# 1140s 481ms/step - loss: 2.0175 - accuracy: 0.4981 - val_loss: 1.3611 - val_accuracy: 0.6356 - lr: 1.0000e-04\n# Epoch 7/30\n# 1085s 457ms/step - loss: 1.9182 - accuracy: 0.5221 - val_loss: 1.3154 - val_accuracy: 0.6446 - lr: 1.0000e-04\n# Epoch 8/30\n# 1081s 456ms/step - loss: 1.8463 - accuracy: 0.5368 - val_loss: 1.2975 - val_accuracy: 0.6541 - lr: 1.0000e-04\n# Epoch 9/30\n# 1077s 454ms/step - loss: 1.7945 - accuracy: 0.5471 - val_loss: 1.2966 - val_accuracy: 0.6568 - lr: 1.0000e-04\n# Epoch 10/30\n# 1074s 453ms/step - loss: 1.7426 - accuracy: 0.5586 - val_loss: 1.2826 - val_accuracy: 0.6523 - lr: 1.0000e-04\n# Epoch 11/30\n# 1057s 446ms/step - loss: 1.6953 - accuracy: 0.5717 - val_loss: 1.2742 - val_accuracy: 0.6547 - lr: 1.0000e-04\n# Epoch 12/30\n# 1076s 454ms/step - loss: 1.6599 - accuracy: 0.5778 - val_loss: 1.2551 - val_accuracy: 0.6623 - lr: 1.0000e-04\n# Epoch 13/30\n# 1076s 454ms/step - loss: 1.6280 - accuracy: 0.5824 - val_loss: 1.2380 - val_accuracy: 0.6653 - lr: 1.0000e-04\n# Epoch 14/30\n# 1077s 454ms/step - loss: 1.5995 - accuracy: 0.5925 - val_loss: 1.2231 - val_accuracy: 0.6684 - lr: 1.0000e-04\n# Epoch 15/30\n# 1078s 454ms/step - loss: 1.5727 - accuracy: 0.5966 - val_loss: 1.2322 - val_accuracy: 0.6647 - lr: 1.0000e-04\n# Epoch 16/30\n# ETA: 0s - loss: 1.5447 - accuracy: 0.6035\n# Epoch 16: ReduceLROnPlateau reducing learning rate to 1.9999999494757503e-05.\n# 1068s 451ms/step - loss: 1.5447 - accuracy: 0.6035 - val_loss: 1.2362 - val_accuracy: 0.6660 - lr: 1.0000e-04\n# Epoch 17/30\n# 1066s 450ms/step - loss: 1.4947 - accuracy: 0.6157 - val_loss: 1.2064 - val_accuracy: 0.6724 - lr: 2.0000e-05\n# Epoch 18/30\n# 1072s 452ms/step - loss: 1.4733 - accuracy: 0.6204 - val_loss: 1.2000 - val_accuracy: 0.6766 - lr: 2.0000e-05\n# Epoch 19/30\n# 1066s 450ms/step - loss: 1.4703 - accuracy: 0.6206 - val_loss: 1.1957 - val_accuracy: 0.6785 - lr: 2.0000e-05\n# Epoch 20/30\n# 1072s 452ms/step - loss: 1.4599 - accuracy: 0.6233 - val_loss: 1.1900 - val_accuracy: 0.6822 - lr: 2.0000e-05\n# Epoch 21/30\n# 1075s 453ms/step - loss: 1.4523 - accuracy: 0.6256 - val_loss: 1.1865 - val_accuracy: 0.6819 - lr: 2.0000e-05\n# Epoch 22/30\n# 1075s 454ms/step - loss: 1.4456 - accuracy: 0.6251 - val_loss: 1.1806 - val_accuracy: 0.6806 - lr: 2.0000e-05\n# Epoch 23/30\n# 1075s 454ms/step - loss: 1.4428 - accuracy: 0.6252 - val_loss: 1.1841 - val_accuracy: 0.6838 - lr: 2.0000e-05\n# Epoch 24/30\n# ETA: 0s - loss: 1.4355 - accuracy: 0.6271\n# Epoch 24: ReduceLROnPlateau reducing learning rate to 3.999999898951501e-06.\n# 1072s 452ms/step - loss: 1.4355 - accuracy: 0.6271 - val_loss: 1.1831 - val_accuracy: 0.6790 - lr: 2.0000e-05\n# Epoch 25/30\n# 1082s 457ms/step - loss: 1.4297 - accuracy: 0.6292 - val_loss: 1.1781 - val_accuracy: 0.6838 - lr: 4.0000e-06\n# Epoch 26/30\n# 1064s 449ms/step - loss: 1.4326 - accuracy: 0.6290 - val_loss: 1.1777 - val_accuracy: 0.6811 - lr: 4.0000e-06\n# Epoch 27/30\n# 1069s 451ms/step - loss: 1.4278 - accuracy: 0.6313 - val_loss: 1.1799 - val_accuracy: 0.6814 - lr: 4.0000e-06\n# Epoch 28/30\n# ETA: 0s - loss: 1.4149 - accuracy: 0.6343\n# Epoch 28: ReduceLROnPlateau reducing learning rate to 7.999999979801942e-07.\n# 1082s 457ms/step - loss: 1.4149 - accuracy: 0.6343 - val_loss: 1.1794 - val_accuracy: 0.6846 - lr: 4.0000e-06\n# Epoch 29/30\n# 1082s 456ms/step - loss: 1.4229 - accuracy: 0.6332 - val_loss: 1.1767 - val_accuracy: 0.6854 - lr: 8.0000e-07\n# Epoch 30/30\n# 1081s 456ms/step - loss: 1.4176 - accuracy: 0.6320 - val_loss: 1.1780 - val_accuracy: 0.6851 - lr: 8.0000e-07\n```\n\n### Model Evaluation\n\n#### Accuracy and Loss\n\n```python\n# the validation accuracy increase keeps slowing while training\ncombine_training_curves(feature_extraction_v2_history, fine_tuning_v2_history, pretraining_epochs=5)\n```\n\n![EfficientNetV2: Smaller Models and Faster Training](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_29.png)\n\n\n#### Predictions\n\n```python\ny_pred = []  # store predicted labels\ny_true = []  # store true labels\n\n# iterate over the dataset\nfor image_batch, label_batch in testing_dataset:   # use dataset.unbatch() with repeat\n   # append true labels\n   y_true.append(label_batch)\n   # compute predictions\n   preds = model_v2.predict(image_batch)\n   # append predicted labels\n   y_pred.append(np.argmax(preds, axis = - 1))\n\n# convert the true and predicted labels into tensors\ncorrect_labels = tf.concat([item for item in y_true], axis = 0)\npredicted_labels = tf.concat([item for item in y_pred], axis = 0)\n```\n\n```python\nplot_confusion_matrix(y_pred=predicted_labels,\n                      y_true=correct_labels,\n                      classes=class_names,\n                      figsize = (88, 88),\n                      text_size=8)\n```\n\n![EfficientNetV2: Smaller Models and Faster Training](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_30.png)\n\n```python\n# Load TensorBoard\n%load_ext tensorboard\n%tensorboard --logdir '../tensorboard/food101/'\n```\n\n![EfficientNetV2: Smaller Models and Faster Training](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_31.png)\n\n![EfficientNetV2: Smaller Models and Faster Training](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_32.png)\n\n```python\n# save the full model\nmodel_v2.save('../saved_models/food101_env2m_30epochs')\n\n# TypeError: Unable to serialize [2.0896919 2.1128857 2.1081853] to JSON. Unrecognized type \u003cclass 'tensorflow.python.framework.ops.EagerTensor'\u003e.\n# tf.__version__ '2.11.0'\n```\n\n## Decreasing Complexity\n\nIn the last experiment I changed 2 parameters:\n\n1. EfficientNet v1 =\u003e EfficientNet v2\n2. EfficientNetB0 =\u003e EfficientNetM\n\nAnd the results I am seeing a very similar. So now I want to try EfficientNetB0 v2 to decrease the complexity of the model and see how this affects the performance.\n\n```python\n# load the version 2 base model\nbase_model_v2_b0 = tf.keras.applications.efficientnet_v2.EfficientNetV2B0(include_top=False)\nbase_model_v2_b0.trainable = False\n```\n\n```python\n# create augmented functional model\ninputs = layers.Input(shape=(IMG_DIM, IMG_DIM, 3), name=\"input_layer\")\nx = data_augmentation_layer_no_rescaling(inputs)\n# normalization not needed for EfficientNet models\n# x = layers.Rescaling(1./255)(x)\n# freeze base layers in inference mode\nx = base_model_v2_b0(x, training=False)\nx = layers.GlobalAveragePooling2D()(x)\nx = layers.Dense(len(class_names))(x)\n# mixed precision requires outputlayer to be of dtype = tf.float32\n# above mixed mixed float16 / below float32 separation\noutputs = layers.Activation(\"softmax\", dtype=tf.float32, name=\"softmax_output_float32\")(x)\nmodel_v2_b0 = tf.keras.Model(inputs, outputs)\n\n# labels are not 1-hot encoded =\u003e sparse_categorical instead of categorical\nmodel_v2_b0.compile(loss=\"sparse_categorical_crossentropy\",\n             optimizer= tf.keras.optimizers.Adam(learning_rate=1e-3),\n             metrics=[\"accuracy\"])\n```\n\n```python\ntf.random.set_seed(SEED)\nfeature_extraction_v2_b0_epochs = 10\n\nfeature_extraction_v2_b0_history = model_v2_b0.fit(training_dataset,\n                                             epochs=feature_extraction_v2_b0_epochs,\n                                             steps_per_epoch=len(training_dataset),\n                                             validation_data=testing_dataset,\n                                             # evaluate performance on 15% of the testing dataset\n                                             validation_steps=int(0.15 * len(testing_dataset)),\n                                             callbacks=[tensorboard_v2_b0_callback,\n                                                        checkpoint_v2_b0_callback,\n                                                        reduce_learning_rate_callback,\n                                                        early_stop_callback])\n\n# Epoch 1/15\n# 665s 275ms/step - loss: 2.6137 - accuracy: 0.3792 - val_loss: 1.6742 - val_accuracy: 0.5556 - lr: 0.0010\n# Epoch 2/10\n# 651s 275ms/step - loss: 2.1238 - accuracy: 0.4749 - val_loss: 1.5503 - val_accuracy: 0.5805 - lr: 0.0010\n# Epoch 3/10\n# 645s 272ms/step - loss: 1.9973 - accuracy: 0.5008 - val_loss: 1.4839 - val_accuracy: 0.6004 - lr: 0.0010\n# Epoch 4/10\n# 624s 263ms/step - loss: 1.9342 - accuracy: 0.5164 - val_loss: 1.4592 - val_accuracy: 0.5985 - lr: 0.0010\n# Epoch 5/10\n# 636s 268ms/step - loss: 1.8739 - accuracy: 0.5274 - val_loss: 1.4295 - val_accuracy: 0.6099 - lr: 0.0010\n# Epoch 6/10\n# 623s 263ms/step - loss: 1.8362 - accuracy: 0.5380 - val_loss: 1.4357 - val_accuracy: 0.6096 - lr: 0.0010\n# Epoch 7/10\n# 570s 240ms/step - loss: 1.8121 - accuracy: 0.5416 - val_loss: 1.3968 - val_accuracy: 0.6200 - lr: 0.0010\n# Epoch 8/10\n# 566s 239ms/step - loss: 1.7965 - accuracy: 0.5447 - val_loss: 1.3965 - val_accuracy: 0.6176 - lr: 0.0010\n# Epoch 9/10\n# 555s 234ms/step - loss: 1.7713 - accuracy: 0.5497 - val_loss: 1.4094 - val_accuracy: 0.6104 - lr: 0.0010\n# Epoch 10/10\n# ETA: 0s - loss: 1.7563 - accuracy: 0.5540\n# Epoch 10: ReduceLROnPlateau reducing learning rate to 0.00020000000949949026.\n# 548s 231ms/step - loss: 1.7562 - accuracy: 0.5540 - val_loss: 1.4167 - val_accuracy: 0.6152 - lr: 0.0010\n\n```\n\n```python\n# unfreeze entire model\nbase_model_v2_b0.trainable = True\n\n# keep only the last 5 layers trainable\nfor layer in base_model_v2_b0.layers[:-5]:\n    layer.trainable = False\n```\n\n```python\n# recompile the model with the new basemodel\nmodel_v2_b0.compile(loss='sparse_categorical_crossentropy',\n               optimizer=tf.keras.optimizers.Adam(learning_rate=1e-4),\n               metrics=['accuracy'])\n```\n\n```python\n# continue training\ntf.random.set_seed(SEED)\nfine_tuning_v2_b0_epochs = feature_extraction_v2_b0_epochs + 25\n\nfine_tuning_v2_b0_history = model_v2_b0.fit(\n                            training_dataset,\n                            epochs=fine_tuning_v2_b0_epochs,\n                            # start from last pre-training checkpoint\n                            # training from epoch 6 - 10\n                            initial_epoch = feature_extraction_v2_b0_history.epoch[-1],\n                            steps_per_epoch=len(training_dataset),\n                            validation_data=testing_dataset,\n                            # evaluate performance on 15% of the testing dataset\n                            validation_steps=int(0.15 * len(testing_dataset)),\n                            callbacks=[tensorboard_v2_b0_callback,\n                                       checkpoint_v2_b0_callback,\n                                       reduce_learning_rate_callback,\n                                       early_stop_callback])\n```\n\n```python\n# save the full model\nmodel_v2_b0.save('../saved_models/food101_env2b0_35epochs')\n```\n\n### Model Training Evaluation\n\n#### Accuracy and Loss\n\n```python\ncombine_training_curves(feature_extraction_v2_b0_history, fine_tuning_v2_b0_history, pretraining_epochs=10)\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_33.png)\n\n\n### Test Prediction Evaluation\n\n```python\n# confusion matrix\ny_pred = []  # store predicted labels\ny_true = []  # store true labels\n\n# iterate over the dataset\nfor image_batch, label_batch in testing_dataset:   # use dataset.unbatch() with repeat\n   # append true labels\n   y_true.append(label_batch)\n   # compute predictions\n   preds = model_v2_b0.predict(image_batch)\n   # append predicted labels\n   y_pred.append(np.argmax(preds, axis = - 1))\n\n# convert the true and predicted labels into tensors\ncorrect_labels = tf.concat([item for item in y_true], axis = 0)\npredicted_labels = tf.concat([item for item in y_pred], axis = 0)\n```\n\n#### Confusion Matrix\n\n```python\nplot_confusion_matrix(y_pred=predicted_labels,\n                      y_true=correct_labels,\n                      classes=class_names,\n                      figsize = (88, 88),\n                      text_size=8)\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_34.png)\n\n\n#### F1 Scores for Labels\n\n```python\n# visualizing the F1 scores per class\nclassification_report_dict = classification_report(y_true=predicted_labels,\n                                                  y_pred=correct_labels,\n                                                  output_dict=True)\n\n\n# extract f1-scores from dictionary\nclass_f1_scores = {}\n\n## loop through classification report\nfor k, v in classification_report_dict.items():\n    # stop when you reach end of table =\u003e class# = accuracy\n    if k == \"accuracy\":\n        break\n    else:\n        # get class name and f1 score for class #\n        class_f1_scores[class_names[int(k)]] = v[\"f1-score\"]\n\n# write it into a dataframe\nf1_scores = pd.DataFrame({\"classname\": list(class_f1_scores.keys()),\n                         \"f1-score\": list(class_f1_scores.values())}).sort_values(\"f1-score\", ascending=False)\n\nprint(f1_scores)\n```\n\n|    | classname | f1-score |\n| -- | -- | -- |\n| 33 |           edamame |  0.972112 |\n| 88 |     seaweed_salad |  0.898129 |\n| 69 |           oysters |  0.895582 |\n| 63 |          macarons |  0.892308 |\n| 65 |           mussels |  0.883333 |\n| .. |               ... |       ... |\n| 22 |  chocolate_mousse |  0.383912 |\n| 39 |         foie_gras |  0.377649 |\n| 15 |           ceviche |  0.364066 |\n| 93 |             steak |  0.333333 |\n| 77 |         pork_chop |  0.316547 |\n\n`[101 rows x 2 columns]`\n\n```python\nf1_scores_inverse = f1_scores.sort_values(by=['f1-score'])\nf1_bar_chart = f1_scores_inverse.plot.barh(x='classname',\n                                  y='f1-score', fontsize=16,\n                                  title=\"F1 Scores vs Class Names\",\n                                  rot=0, legend=True,\n                                  figsize=(12,36))\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_35.png)\n\n\n#### Find Wrong Predictions with Highest Confidence\n\n```python\n# making predictions on all 25250 validation images for 101 classes\ntest_prediction_probabilities = model_v2_b0.predict(testing_dataset, verbose=1)\nprint(test_prediction_probabilities.shape)\n# (25250, 101)\n```\n\n```python\n# find false prediction that have the highest confidence\nprediction_quality = pd.DataFrame({\"y_true\": correct_labels,\n                                  \"y_pred\": predicted_labels,\n                                  \"pred_conf\": test_prediction_probabilities.max(axis=1),\n                                  \"y_true_classname\": [class_names[i] for i in correct_labels],\n                                  \"y_pred_classname\": [class_names[i] for i in predicted_labels]})\n\nprediction_quality\n```\n\n|    | y_true | y_pred | pred_conf | y_true_classname | y_pred_classname |\n| -- |   --   |  --    |    --     |       --         |       --         |\n| 0 | \t78 | \t8 | \t0.679352 | \tpoutine | \tbread_pudding |\n| 1 | \t100 | \t100 | \t0.849204 | \twaffles | \twaffles |\n| 2 | \t79 | \t79 | \t0.866372 | \tprime_rib | \tprime_rib |\n| 3 | \t4 | \t4 | \t0.803154 | \tbeef_tartare | \tbeef_tartare |\n| 4 | \t37 | \t42 | \t0.990849 | \tfilet_mignon | \tfrench_toast |\n| ... | \t... | \t... | \t... | \t... | \t... |\n| 25245 | \t53 | \t53 | \t0.700280 | \thamburger | \thamburger |\n| 25246 | \t13 | \t13 | \t0.944595 | \tcaprese_salad | \tcaprese_salad |\n| 25247 | \t53 | \t53 | \t0.362593 | \thamburger | \thamburger |\n| 25248 | \t11 | \t11 | \t0.998997 | \tcaesar_salad | \tcaesar_salad |\n| 25249 | \t87 | \t87 | \t0.985820 | \tscallops | \tscallops |\n\n`25250 rows × 5 columns`\n\n```python\n# add bool comlumn for correct predictions\nprediction_quality[\"pred_correct\"] = prediction_quality[\"y_true\"] == prediction_quality[\"y_pred\"]\n```\n\n```python\n# create new dataframe with the 100 most wrong predictions\ntop_100_wrong = prediction_quality[prediction_quality[\"pred_correct\"] == False].sort_values(\"pred_conf\", ascending=False)[:100]\ntop_100_wrong\n```\n\n| \t    | y_true | y_pred |\tpred_conf | y_true_classname | y_pred_classname | pred_correct |\n| -- | \t-- | -- | \t-- | \t-- | --  | \t-- |\n| 21045 | \t66 | \t67 | \t1.000000 | \tnachos | \tomelette | \tFalse |\n| 610 | \t7 | \t41 | \t1.000000 | \tbibimbap | \tfrench_onion_soup | \tFalse |\n| 15468 | \t68 | \t96 | \t1.000000 | \tonion_rings | \ttacos | \tFalse |\n| 24295 | \t35 | \t86 | \t1.000000 | \tescargots | \tsashimi | \tFalse |\n| 20157 | \t42 | \t50 | \t1.000000 | \tfrench_toast | \tgrilled_salmon | \tFalse |\n| ... | \t... | \t... | \t... | \t... | \t... | \t... |\n| 12150 | \t89 | \t0 | \t0.999972 | \tshrimp_and_grits | \tapple_pie | \tFalse |\n| 15385 | \t95 | \t61 | \t0.999972 | \tsushi | \tlobster_roll_sandwich | \tFalse |\n| 10664 | \t71 | \t67 | \t0.999972 | \tpaella | \tomelette | \tFalse |\n| 23552 | \t89 | \t18 | \t0.999969 | \tshrimp_and_grits | \tchicken_curry | \tFalse |\n| 8587 | \t85 | \t0 | \t0.999969 | \tsamosa | \tapple_pie | \tFalse |\n\n```python\n# what predictions are most often wrong\ngrouped_top_100_wrong_pred = top_100_wrong.groupby(['y_pred', 'y_pred_classname']).agg(', '.join).reset_index()\npd.set_option('display.max_colwidth', None)\ngrouped_top_100_wrong_pred[:50]\n```\n\n|   | y_pred | y_pred_classname | y_true_classname |\n| -- | -- | -- | -- |\n| 0 |\t0 \t | apple_pie | pancakes, miso_soup, risotto, grilled_cheese_sandwich, shrimp_and_grits, samosa |\n| 1 | 1 | baby_back_ribs |\tsteak |\n| 2 | 3 | beef_carpaccio |\tbeef_tartare, caesar_salad |\n| 3 | 5 | beet_salad |\tbeef_carpaccio, deviled_eggs |\n| 4 | 6 | beignets |\travioli |\n| 5 | 8 | bread_pudding |\tpork_chop, strawberry_shortcake, strawberry_shortcake, beet_salad |\n| 6 | 9 | breakfast_burrito |\thuevos_rancheros, lasagna, chicken_quesadilla, omelette |\n| 7 | 10 | bruschetta |\ttuna_tartare, huevos_rancheros |\n| 8 | 12 | cannoli |\ttuna_tartare, carrot_cake |\n| 9 | 15 | ceviche |\tbeet_salad |\n| 10 | 17 | cheese_plate |\tsashimi, grilled_cheese_sandwich |\n| 11 | 18 | chicken_curry |\travioli, shrimp_and_grits |\n| 12 | 20 | chicken_wings |\tfrench_fries, peking_duck |\n| 13 | 21 | chocolate_cake |\tchocolate_mousse, chocolate_mousse, grilled_salmon |\n| 14 | 22 | chocolate_mousse |\tapple_pie |\n| 15 | 26 | crab_cakes |\tfoie_gras |\n| 16 | 27 | creme_brulee |\tpancakes |\n| 17 | 37 | filet_mignon |\tsteak, tuna_tartare, chocolate_cake, prime_rib |\n| 18 | 38 | fish_and_chips |\tpulled_pork_sandwich |\n| 19 | 39 | foie_gras |\tapple_pie, shrimp_and_grits |\n| 20 | 41 | french_onion_soup |\tbibimbap |\n| 21 | 42 | french_toast |\tcheesecake, huevos_rancheros, churros, waffles |\n| 22 | 45 | frozen_yogurt |\tbaklava |\n| 23 | 46 | garlic_bread |\tdumplings |\n| 24 | 47 | gnocchi |\travioli |\n| 25 | 48 | greek_salad |\tcaesar_salad |\n| 26 | 49 | grilled_cheese_sandwich |\tchicken_quesadilla, hummus, bruschetta, garlic_bread |\n| 27 | 50 | grilled_salmon |\tfrench_toast, crab_cakes |\n| 28 | 52 | gyoza |\tchicken_quesadilla, grilled_cheese_sandwich |\n| 29 | 53 | hamburger |\tclub_sandwich, onion_rings |\n| 30 | 55 | hot_dog |\ttacos |\n| 31 | 56 | huevos_rancheros |\tomelette |\n| 32 | 58 | ice_cream |\tmacaroni_and_cheese, frozen_yogurt |\n| 33 | 60 | lobster_bisque |\tpeking_duck |\n| 34 | 61 | lobster_roll_sandwich |\tsushi |\n| 35 | 64 | miso_soup |\tfrench_onion_soup |\n| 36 | 67 | omelette |\tnachos, caesar_salad, paella |\n| 37 | 74 | peking_duck |\tfrozen_yogurt, oysters |\n| 38 | 77 | pork_chop |\tchicken_wings |\n| 39 | 79 | prime_rib |\tbaby_back_ribs, steak |\n| 40 | 81 | ramen |\tpho |\n| 41 | 82 | ravioli |\tgnocchi, gnocchi |\n| 42 | 83 | red_velvet_cake |\tstrawberry_shortcake, panna_cotta |\n| 43 | 84 | risotto |\travioli |\n| 44 | 85 | samosa |\ttuna_tartare, breakfast_burrito |\n| 45 | 86 | sashimi |\tescargots, strawberry_shortcake, hummus |\n| 46 | 91 | spaghetti_carbonara |\tspaghetti_bolognese |\n| 47 | 92 | spring_rolls |\tfish_and_chips, huevos_rancheros |\n| 48 | 93 | steak |\tbaby_back_ribs, bread_pudding, foie_gras |\n| 49 | 96 | tacos |\tonion_rings |\n| 50 | 97 | takoyaki |\tspaghetti_bolognese |\n| 51 | 98 | tiramisu |\tcannoli, chocolate_mousse |\n| 52 | 100 | waffles |\tnachos |\n\n```python\n# what classes cause the most wrong predictions\ngrouped_top_100_wrong_cause = top_100_wrong.groupby(['y_true', 'y_true_classname']).agg(', '.join).reset_index()\ngrouped_top_100_wrong_cause[:50]\n```\n\n|   | y_true | y_true_classname | y_pred_classname |\n| -- | -- | -- | -- |\n| 0 | 0 | apple_pie | foie_gras, chocolate_mousse |\n| 1 | 1 | baby_back_ribs | prime_rib, steak |\n| 2 | 2 | baklava | frozen_yogurt |\n| 3 | 3 | beef_carpaccio | beet_salad |\n| 4 | 4 | beef_tartare | beef_carpaccio |\n| 5 | 5 | beet_salad | ceviche, bread_pudding |\n| 6 | 7 | bibimbap | french_onion_soup |\n| 7 | 8 | bread_pudding | steak |\n| 8 | 9 | breakfast_burrito | samosa |\n| 9 | 10 | bruschetta | grilled_cheese_sandwich |\n| 10 | 11 | caesar_salad | omelette, greek_salad, beef_carpaccio |\n| 11 | 12 | cannoli | tiramisu |\n| 12 | 14 | carrot_cake | cannoli |\n| 13 | 16 | cheesecake | french_toast |\n| 14 | 19 | chicken_quesadilla | grilled_cheese_sandwich, breakfast_burrito, gyoza |\n| 15 | 20 | chicken_wings | pork_chop |\n| 16 | 21 | chocolate_cake | filet_mignon |\n| 17 | 22 | chocolate_mousse | chocolate_cake, chocolate_cake, tiramisu |\n| 18 | 23 | churros | french_toast |\n| 19 | 25 | club_sandwich | hamburger |\n| 20 | 26 | crab_cakes | grilled_salmon |\n| 21 | 30 | deviled_eggs | beet_salad |\n| 22 | 32 | dumplings | garlic_bread |\n| 23 | 35 | escargots | sashimi |\n| 24 | 38 | fish_and_chips | spring_rolls |\n| 25 | 39 | foie_gras | crab_cakes, steak |\n| 26 | 40 | french_fries | chicken_wings |\n| 27 | 41 | french_onion_soup | miso_soup |\n| 28 | 42 | french_toast | grilled_salmon |\n| 29 | 45 | frozen_yogurt | peking_duck, ice_cream |\n| 30 | 46 | garlic_bread | grilled_cheese_sandwich |\n| 31 | 47 | gnocchi | ravioli, ravioli |\n| 32 | 49 | grilled_cheese_sandwich | cheese_plate, apple_pie, gyoza |\n| 33 | 50 | grilled_salmon | chocolate_cake |\n| 34 | 56 | huevos_rancheros | bruschetta, french_toast, breakfast_burrito, spring_rolls |\n| 35 | 57 | hummus | grilled_cheese_sandwich, sashimi |\n| 36 | 59 | lasagna | breakfast_burrito |\n| 37 | 62 | macaroni_and_cheese | ice_cream |\n| 38 | 64 | miso_soup | apple_pie |\n| 39 | 66 | nachos | omelette, waffles |\n| 40 | 67 | omelette | huevos_rancheros, breakfast_burrito |\n| 41 | 68 | onion_rings | tacos, hamburger |\n| 42 | 69 | oysters | peking_duck |\n| 43 | 71 | paella | omelette |\n| 44 | 72 | pancakes | apple_pie, creme_brulee |\n| 45 | 73 | panna_cotta | red_velvet_cake |\n| 46 | 74 | peking_duck | lobster_bisque, chicken_wings |\n| 47 | 75 | pho | ramen |\n| 48 | 77 | pork_chop | bread_pudding |\n| 49 | 79 | prime_rib | filet_mignon |\n\n\n### Run Custom Predictions\n\n```python\n# get list of custom image file paths\ncustom_images_path = \"../datasets/custom_images/\"\ncustom_images = [ custom_images_path + img_path for img_path in os.listdir(custom_images_path)]\ncustom_images\n\n# ['../datasets/custom_images/cheesecake.jpg',\n#  '../datasets/custom_images/crema_catalana.jpg',\n#  '../datasets/custom_images/fish_and_chips.jpg',\n#  '../datasets/custom_images/jiaozi.jpg',\n#  '../datasets/custom_images/paella.jpg',\n#  '../datasets/custom_images/pho.jpg',\n#  '../datasets/custom_images/quesadilla.jpg',\n#  '../datasets/custom_images/ravioli.jpg',\n#  '../datasets/custom_images/waffles.jpg']\n```\n\n```python\n# run prediction on custom images\nfor image in custom_images:\n    image = load_and_preprocess_image(image, normalize=False)\n    # test image is (224, 224, 3) but model expects batch shape (None, 224, 224, 3)\n    image_expanded = tf.expand_dims(image, axis=0)\n    # get probabilities over all classes\n    prediction_probabilities = model_v2_b0.predict(image_expanded)\n    # get classname for highest probability\n    predicted_class =  class_names[prediction_probabilities.argmax()]\n    # plot normalized image\n    plt.figure()\n    plt.imshow(image/255.)\n    plt.title(f\"Pred: {predicted_class} ({prediction_probabilities.max()*100:.2f} %)\")\n    plt.axis(False)\n```\n\n![Keras Applications \u0026 Tensorflow Datasets](https://github.com/mpolinowski/keras_transfer_learning_2023/blob/master/assets/04_Tensorflow_Transfer_Learning_36.png)\n\n```python\n\n```\n","project_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpolinowski%2Fkeras_transfer_learning_2023","html_url":"https://awesome.ecosyste.ms/projects/github.com%2Fmpolinowski%2Fkeras_transfer_learning_2023","lists_url":"https://awesome.ecosyste.ms/api/v1/projects/github.com%2Fmpolinowski%2Fkeras_transfer_learning_2023/lists"}