11_training_deep_neural_networks_q.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> 8. CIFAR10 이미지 데이터셋에 심층 신경망을 훈련해보세요. \\\n",
    "> a. 100개의 뉴런을 가진 은닉층 20개로 심층 신경망을 만들어보세요. He 초기화와 ELU 활성화 함수를 사용하세요."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz\n",
      "170498071/170498071 [==============================] - 343s 2us/step\n"
     ]
    }
   ],
   "source": [
    "from keras import datasets\n",
    "\n",
    "(X_train_full, y_train_full), (X_test, y_test) = datasets.cifar10.load_data()\n",
    "\n",
    "X_train = X_train_full[5000:]\n",
    "y_train = y_train_full[5000:]\n",
    "X_valid = X_train_full[:5000]\n",
    "y_valid = y_train_full[:5000]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(45000, 32, 32, 3)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "import keras \n",
    "\n",
    "model=keras.models.Sequential([\n",
    "    keras.layers.Flatten(input_shape=[X_train.shape[1],X_train.shape[2],X_train.shape[3]]),\n",
    "])\n",
    "\n",
    "for _ in range(20):\n",
    "    model.add(keras.layers.Dense(100, activation=\"elu\",\n",
    "                                 kernel_initializer=\"he_normal\"))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> b. Nadam 옵티마이저, 조기 종료 사용한 네트워크 훈련 \\\n",
    "keras.datasets.cifar10.load_data()를 사용하여 데이터 적재가능   \n",
    "10개 클래스, 32x32 크기, 컬러 이미지 60,000개 구성 \\\n",
    "10개 뉴런, 소프트 맥스 활성화 함수 출력층 필요.  \\\n",
    "적절한 학습률 .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recall(y_target, y_pred):\n",
    "    # clip(t, clip_value_min, clip_value_max) : clip_value_min~clip_value_max 이외 가장자리를 깎아 낸다\n",
    "    # round : 반올림한다\n",
    "    y_target_yn = K.round(K.clip(y_target, 0, 1)) # 실제값을 0(Negative) 또는 1(Positive)로 설정한다\n",
    "    y_pred_yn = K.round(K.clip(y_pred, 0, 1)) # 예측값을 0(Negative) 또는 1(Positive)로 설정한다\n",
    "\n",
    "    # True Positive는 실제 값과 예측 값이 모두 1(Positive)인 경우이다\n",
    "    count_true_positive = K.sum(y_target_yn * y_pred_yn) \n",
    "\n",
    "    # (True Positive + False Negative) = 실제 값이 1(Positive) 전체\n",
    "    count_true_positive_false_negative = K.sum(y_target_yn)\n",
    "\n",
    "    # Recall =  (True Positive) / (True Positive + False Negative)\n",
    "    # K.epsilon()는 'divide by zero error' 예방차원에서 작은 수를 더한다\n",
    "    recall = count_true_positive / (count_true_positive_false_negative + K.epsilon())\n",
    "\n",
    "    # return a single tensor value\n",
    "    return recall\n",
    "\n",
    "\n",
    "def precision(y_target, y_pred):\n",
    "    # clip(t, clip_value_min, clip_value_max) : clip_value_min~clip_value_max 이외 가장자리를 깎아 낸다\n",
    "    # round : 반올림한다\n",
    "    y_pred_yn = K.round(K.clip(y_pred, 0, 1)) # 예측값을 0(Negative) 또는 1(Positive)로 설정한다\n",
    "    y_target_yn = K.round(K.clip(y_target, 0, 1)) # 실제값을 0(Negative) 또는 1(Positive)로 설정한다\n",
    "\n",
    "    # True Positive는 실제 값과 예측 값이 모두 1(Positive)인 경우이다\n",
    "    count_true_positive = K.sum(y_target_yn * y_pred_yn) \n",
    "\n",
    "    # (True Positive + False Positive) = 예측 값이 1(Positive) 전체\n",
    "    count_true_positive_false_positive = K.sum(y_pred_yn)\n",
    "\n",
    "    # Precision = (True Positive) / (True Positive + False Positive)\n",
    "    # K.epsilon()는 'divide by zero error' 예방차원에서 작은 수를 더한다\n",
    "    precision = count_true_positive / (count_true_positive_false_positive + K.epsilon())\n",
    "    return precision\n",
    "\n",
    "def f1score(y_target, y_pred):\n",
    "    _recall = recall(y_target, y_pred)\n",
    "    _precision = precision(y_target, y_pred)\n",
    "    _f1score = ( 2 * _recall * _precision) / (_recall + _precision+ K.epsilon())\n",
    "    return _f1score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.add(keras.layers.Dense(10, activation=\"softmax\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer=keras.optimizers.Nadam(learning_rate=5e-5)\n",
    "model.compile(loss=\"sparse_categorical_crossentropy\",\n",
    "              optimizer=optimizer,\n",
    "              metrics=[\"accuracy\",f1score])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "early_stopping_cb = keras.callbacks.EarlyStopping(patience=20)\n",
    "model_checkpoint_cb = keras.callbacks.ModelCheckpoint(\"my_cifar10_model.h5\", save_best_only=True)\n",
    "run_index = 1 # increment every time you train the model\n",
    "run_logdir = os.path.join(os.curdir, \"my_cifar10_logs\", \"run_{:03d}\".format(run_index))\n",
    "tensorboard_cb = keras.callbacks.TensorBoard(run_logdir)\n",
    "callbacks = [early_stopping_cb, model_checkpoint_cb, tensorboard_cb]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tensorboard in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (2.11.0)\n",
      "Requirement already satisfied: werkzeug>=1.0.1 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (2.2.2)\n",
      "Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (0.6.1)\n",
      "Requirement already satisfied: requests<3,>=2.21.0 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (2.28.1)\n",
      "Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (1.8.1)\n",
      "Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (0.4.6)\n",
      "Requirement already satisfied: setuptools>=41.0.0 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (63.4.1)\n",
      "Requirement already satisfied: google-auth<3,>=1.6.3 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (2.14.1)\n",
      "Requirement already satisfied: wheel>=0.26 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (0.37.1)\n",
      "Requirement already satisfied: markdown>=2.6.8 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (3.4.1)\n",
      "Requirement already satisfied: protobuf<4,>=3.9.2 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (3.19.6)\n",
      "Requirement already satisfied: absl-py>=0.4 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (1.3.0)\n",
      "Requirement already satisfied: grpcio>=1.24.3 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (1.50.0)\n",
      "Requirement already satisfied: numpy>=1.12.0 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from tensorboard) (1.21.6)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard) (4.9)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard) (0.2.8)\n",
      "Requirement already satisfied: cachetools<6.0,>=2.0.0 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard) (5.2.0)\n",
      "Requirement already satisfied: six>=1.9.0 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from google-auth<3,>=1.6.3->tensorboard) (1.16.0)\n",
      "Requirement already satisfied: requests-oauthlib>=0.7.0 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard) (1.3.1)\n",
      "Requirement already satisfied: importlib-metadata>=4.4 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from markdown>=2.6.8->tensorboard) (4.12.0)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard) (3.4)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard) (2022.9.24)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard) (1.26.12)\n",
      "Requirement already satisfied: charset-normalizer<3,>=2 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from requests<3,>=2.21.0->tensorboard) (2.1.1)\n",
      "Requirement already satisfied: MarkupSafe>=2.1.1 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from werkzeug>=1.0.1->tensorboard) (2.1.1)\n",
      "Requirement already satisfied: zipp>=0.5 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard) (3.8.1)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard) (4.3.0)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard) (0.4.8)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /Users/gahyunson/opt/anaconda3/envs/mywork/lib/python3.7/site-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard) (3.2.2)\n"
     ]
    }
   ],
   "source": [
    "!pip install tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext tensorboard"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "\n",
       "      <iframe id=\"tensorboard-frame-cbdf6cabf2a67200\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
       "      </iframe>\n",
       "      <script>\n",
       "        (function() {\n",
       "          const frame = document.getElementById(\"tensorboard-frame-cbdf6cabf2a67200\");\n",
       "          const url = new URL(\"/\", window.location);\n",
       "          const port = 6006;\n",
       "          if (port) {\n",
       "            url.port = port;\n",
       "          }\n",
       "          frame.src = url;\n",
       "        })();\n",
       "      </script>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%tensorboard --logdir=./my_cifar10_logs --port=6006"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "K = keras.backend"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/100\n",
      "1407/1407 [==============================] - 28s 15ms/step - loss: 4.1481 - accuracy: 0.1752 - f1score: 0.2633 - val_loss: 2.1532 - val_accuracy: 0.2204 - val_f1score: 0.1592\n",
      "Epoch 2/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 2.0392 - accuracy: 0.2520 - f1score: 0.1345 - val_loss: 2.0920 - val_accuracy: 0.2470 - val_f1score: 0.2974\n",
      "Epoch 3/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.9202 - accuracy: 0.3010 - f1score: 0.1817 - val_loss: 1.9050 - val_accuracy: 0.3080 - val_f1score: 0.2389\n",
      "Epoch 4/100\n",
      "1407/1407 [==============================] - 17s 12ms/step - loss: 1.8505 - accuracy: 0.3290 - f1score: 0.2301 - val_loss: 1.8731 - val_accuracy: 0.3222 - val_f1score: 0.3340\n",
      "Epoch 5/100\n",
      "1407/1407 [==============================] - 17s 12ms/step - loss: 1.7947 - accuracy: 0.3464 - f1score: 0.2751 - val_loss: 1.8017 - val_accuracy: 0.3508 - val_f1score: 0.3208\n",
      "Epoch 6/100\n",
      "1407/1407 [==============================] - 17s 12ms/step - loss: 1.7504 - accuracy: 0.3678 - f1score: 0.3060 - val_loss: 1.7329 - val_accuracy: 0.3768 - val_f1score: 0.2856\n",
      "Epoch 7/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.7071 - accuracy: 0.3810 - f1score: 0.3377 - val_loss: 1.7052 - val_accuracy: 0.3850 - val_f1score: 0.3394\n",
      "Epoch 8/100\n",
      "1407/1407 [==============================] - 27s 19ms/step - loss: 1.6721 - accuracy: 0.3979 - f1score: 0.3707 - val_loss: 1.6544 - val_accuracy: 0.3994 - val_f1score: 0.3981\n",
      "Epoch 9/100\n",
      "1407/1407 [==============================] - 27s 19ms/step - loss: 1.6404 - accuracy: 0.4094 - f1score: 0.3957 - val_loss: 1.6875 - val_accuracy: 0.3914 - val_f1score: 0.4247\n",
      "Epoch 10/100\n",
      "1407/1407 [==============================] - 21s 15ms/step - loss: 1.6146 - accuracy: 0.4173 - f1score: 0.4182 - val_loss: 1.6580 - val_accuracy: 0.4046 - val_f1score: 0.4158\n",
      "Epoch 11/100\n",
      "1407/1407 [==============================] - 23s 16ms/step - loss: 1.5918 - accuracy: 0.4247 - f1score: 0.4435 - val_loss: 1.5958 - val_accuracy: 0.4270 - val_f1score: 0.4291\n",
      "Epoch 12/100\n",
      "1407/1407 [==============================] - 23s 16ms/step - loss: 1.5665 - accuracy: 0.4358 - f1score: 0.4575 - val_loss: 1.6053 - val_accuracy: 0.4236 - val_f1score: 0.4620\n",
      "Epoch 13/100\n",
      "1407/1407 [==============================] - 25s 18ms/step - loss: 1.5464 - accuracy: 0.4442 - f1score: 0.4695 - val_loss: 1.5960 - val_accuracy: 0.4330 - val_f1score: 0.4622\n",
      "Epoch 14/100\n",
      "1407/1407 [==============================] - 26s 18ms/step - loss: 1.5280 - accuracy: 0.4486 - f1score: 0.4897 - val_loss: 1.6749 - val_accuracy: 0.3978 - val_f1score: 0.5329\n",
      "Epoch 15/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.5088 - accuracy: 0.4570 - f1score: 0.4966 - val_loss: 1.6423 - val_accuracy: 0.4198 - val_f1score: 0.4939\n",
      "Epoch 16/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.4963 - accuracy: 0.4630 - f1score: 0.5074 - val_loss: 1.5813 - val_accuracy: 0.4328 - val_f1score: 0.4433\n",
      "Epoch 17/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.4751 - accuracy: 0.4698 - f1score: 0.5174 - val_loss: 1.6121 - val_accuracy: 0.4262 - val_f1score: 0.5540\n",
      "Epoch 18/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.4619 - accuracy: 0.4742 - f1score: 0.5297 - val_loss: 1.5436 - val_accuracy: 0.4514 - val_f1score: 0.5397\n",
      "Epoch 19/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.4468 - accuracy: 0.4795 - f1score: 0.5384 - val_loss: 1.5883 - val_accuracy: 0.4368 - val_f1score: 0.5354\n",
      "Epoch 20/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.4338 - accuracy: 0.4825 - f1score: 0.5510 - val_loss: 1.5203 - val_accuracy: 0.4632 - val_f1score: 0.5177\n",
      "Epoch 21/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.4188 - accuracy: 0.4880 - f1score: 0.5563 - val_loss: 1.5663 - val_accuracy: 0.4362 - val_f1score: 0.5371\n",
      "Epoch 22/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.4078 - accuracy: 0.4929 - f1score: 0.5649 - val_loss: 1.5111 - val_accuracy: 0.4636 - val_f1score: 0.5241\n",
      "Epoch 23/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.3973 - accuracy: 0.4950 - f1score: 0.5662 - val_loss: 1.5349 - val_accuracy: 0.4502 - val_f1score: 0.5743\n",
      "Epoch 24/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.3830 - accuracy: 0.5021 - f1score: 0.5753 - val_loss: 1.4872 - val_accuracy: 0.4682 - val_f1score: 0.5388\n",
      "Epoch 25/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.3723 - accuracy: 0.5056 - f1score: 0.5837 - val_loss: 1.4995 - val_accuracy: 0.4636 - val_f1score: 0.5516\n",
      "Epoch 26/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.3577 - accuracy: 0.5128 - f1score: 0.5924 - val_loss: 1.4907 - val_accuracy: 0.4632 - val_f1score: 0.5566\n",
      "Epoch 27/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.3446 - accuracy: 0.5174 - f1score: 0.6017 - val_loss: 1.5102 - val_accuracy: 0.4670 - val_f1score: 0.5753\n",
      "Epoch 28/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.3382 - accuracy: 0.5187 - f1score: 0.6064 - val_loss: 1.4968 - val_accuracy: 0.4672 - val_f1score: 0.5436\n",
      "Epoch 29/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.3262 - accuracy: 0.5232 - f1score: 0.6097 - val_loss: 1.5338 - val_accuracy: 0.4666 - val_f1score: 0.5873\n",
      "Epoch 30/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.3153 - accuracy: 0.5266 - f1score: 0.6155 - val_loss: 1.4803 - val_accuracy: 0.4756 - val_f1score: 0.6213\n",
      "Epoch 31/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.3060 - accuracy: 0.5298 - f1score: 0.6208 - val_loss: 1.5342 - val_accuracy: 0.4614 - val_f1score: 0.6126\n",
      "Epoch 32/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.2965 - accuracy: 0.5323 - f1score: 0.6240 - val_loss: 1.5168 - val_accuracy: 0.4692 - val_f1score: 0.6143\n",
      "Epoch 33/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.2868 - accuracy: 0.5366 - f1score: 0.6329 - val_loss: 1.5126 - val_accuracy: 0.4700 - val_f1score: 0.6176\n",
      "Epoch 34/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.2786 - accuracy: 0.5412 - f1score: 0.6360 - val_loss: 1.5313 - val_accuracy: 0.4626 - val_f1score: 0.6085\n",
      "Epoch 35/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.2669 - accuracy: 0.5455 - f1score: 0.6420 - val_loss: 1.4944 - val_accuracy: 0.4776 - val_f1score: 0.6093\n",
      "Epoch 36/100\n",
      "1407/1407 [==============================] - 18s 13ms/step - loss: 1.2615 - accuracy: 0.5482 - f1score: 0.6488 - val_loss: 1.4879 - val_accuracy: 0.4748 - val_f1score: 0.6047\n",
      "Epoch 37/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.2508 - accuracy: 0.5512 - f1score: 0.6492 - val_loss: 1.5044 - val_accuracy: 0.4732 - val_f1score: 0.6269\n",
      "Epoch 38/100\n",
      "1407/1407 [==============================] - 20s 15ms/step - loss: 1.2387 - accuracy: 0.5574 - f1score: 0.6585 - val_loss: 1.5510 - val_accuracy: 0.4576 - val_f1score: 0.6429\n",
      "Epoch 39/100\n",
      "1407/1407 [==============================] - 22s 15ms/step - loss: 1.2342 - accuracy: 0.5568 - f1score: 0.6555 - val_loss: 1.4682 - val_accuracy: 0.4824 - val_f1score: 0.6255\n",
      "Epoch 40/100\n",
      "1407/1407 [==============================] - 23s 16ms/step - loss: 1.2262 - accuracy: 0.5605 - f1score: 0.6581 - val_loss: 1.5354 - val_accuracy: 0.4748 - val_f1score: 0.6515\n",
      "Epoch 41/100\n",
      "1407/1407 [==============================] - 19s 14ms/step - loss: 1.2175 - accuracy: 0.5613 - f1score: 0.6654 - val_loss: 1.5189 - val_accuracy: 0.4752 - val_f1score: 0.6411\n",
      "Epoch 42/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.2063 - accuracy: 0.5668 - f1score: 0.6727 - val_loss: 1.5647 - val_accuracy: 0.4580 - val_f1score: 0.6654\n",
      "Epoch 43/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.2003 - accuracy: 0.5689 - f1score: 0.6772 - val_loss: 1.5231 - val_accuracy: 0.4754 - val_f1score: 0.6493\n",
      "Epoch 44/100\n",
      "1407/1407 [==============================] - 19s 13ms/step - loss: 1.1889 - accuracy: 0.5739 - f1score: 0.6783 - val_loss: 1.5503 - val_accuracy: 0.4712 - val_f1score: 0.6467\n",
      "Epoch 45/100\n",
      "1407/1407 [==============================] - 19s 14ms/step - loss: 1.1836 - accuracy: 0.5762 - f1score: 0.6828 - val_loss: 1.5209 - val_accuracy: 0.4770 - val_f1score: 0.6557\n",
      "Epoch 46/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.1762 - accuracy: 0.5779 - f1score: 0.6846 - val_loss: 1.4951 - val_accuracy: 0.4814 - val_f1score: 0.6541\n",
      "Epoch 47/100\n",
      "1407/1407 [==============================] - 19s 14ms/step - loss: 1.1685 - accuracy: 0.5789 - f1score: 0.6883 - val_loss: 1.5223 - val_accuracy: 0.4830 - val_f1score: 0.6492\n",
      "Epoch 48/100\n",
      "1407/1407 [==============================] - 21s 15ms/step - loss: 1.1604 - accuracy: 0.5835 - f1score: 0.6917 - val_loss: 1.5253 - val_accuracy: 0.4884 - val_f1score: 0.6669\n",
      "Epoch 49/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.1545 - accuracy: 0.5839 - f1score: 0.6946 - val_loss: 1.5329 - val_accuracy: 0.4870 - val_f1score: 0.6680\n",
      "Epoch 50/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.1448 - accuracy: 0.5880 - f1score: 0.6987 - val_loss: 1.5432 - val_accuracy: 0.4734 - val_f1score: 0.6680\n",
      "Epoch 51/100\n",
      "1407/1407 [==============================] - 19s 14ms/step - loss: 1.1404 - accuracy: 0.5894 - f1score: 0.7023 - val_loss: 1.5365 - val_accuracy: 0.4886 - val_f1score: 0.6650\n",
      "Epoch 52/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.1321 - accuracy: 0.5934 - f1score: 0.7021 - val_loss: 1.6308 - val_accuracy: 0.4594 - val_f1score: 0.6865\n",
      "Epoch 53/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.1244 - accuracy: 0.5978 - f1score: 0.7071 - val_loss: 1.5196 - val_accuracy: 0.4882 - val_f1score: 0.6785\n",
      "Epoch 54/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.1194 - accuracy: 0.6011 - f1score: 0.7096 - val_loss: 1.5188 - val_accuracy: 0.4890 - val_f1score: 0.6651\n",
      "Epoch 55/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.1121 - accuracy: 0.5982 - f1score: 0.7121 - val_loss: 1.5625 - val_accuracy: 0.4840 - val_f1score: 0.6722\n",
      "Epoch 56/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.1035 - accuracy: 0.6028 - f1score: 0.7174 - val_loss: 1.5595 - val_accuracy: 0.4846 - val_f1score: 0.6849\n",
      "Epoch 57/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.0948 - accuracy: 0.6082 - f1score: 0.7163 - val_loss: 1.5618 - val_accuracy: 0.4874 - val_f1score: 0.6922\n",
      "Epoch 58/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.0922 - accuracy: 0.6060 - f1score: 0.7163 - val_loss: 1.5740 - val_accuracy: 0.4786 - val_f1score: 0.6907\n",
      "Epoch 59/100\n",
      "1407/1407 [==============================] - 20s 14ms/step - loss: 1.0854 - accuracy: 0.6100 - f1score: 0.7231 - val_loss: 1.5831 - val_accuracy: 0.4836 - val_f1score: 0.6827\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x7fd05a0a31d0>"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# X_means = X_train.mean(axis=0)\n",
    "# X_stds = X_train.std(axis=0)\n",
    "# X_train_scaled = (X_train - X_means) / X_stds\n",
    "# X_valid_scaled = (X_valid - X_means) / X_stds\n",
    "# X_test_scaled = (X_test - X_means) / X_stds\n",
    "\n",
    "model.fit(X_train, y_train, epochs=100,\n",
    "          validation_data=(X_valid, y_valid),\n",
    "          callbacks=callbacks)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "Unknown metric function: 'f1score'. Please ensure you are using a `keras.utils.custom_object_scope` and that this object is included in the scope. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m/var/folders/0s/knzc6bzs1l9dgw6x4k24t9740000gn/T/ipykernel_5182/985072887.py\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mmodel\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mkeras\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mmodels\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_model\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"my_cifar10_model.h5\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX_valid\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0my_valid\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/keras/utils/traceback_utils.py\u001b[0m in \u001b[0;36merror_handler\u001b[0;34m(*args, **kwargs)\u001b[0m\n\u001b[1;32m     68\u001b[0m             \u001b[0;31m# To get the full stack trace, call:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m             \u001b[0;31m# `tf.debugging.disable_traceback_filtering()`\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 70\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mwith_traceback\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mfiltered_tb\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     71\u001b[0m         \u001b[0;32mfinally\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     72\u001b[0m             \u001b[0;32mdel\u001b[0m \u001b[0mfiltered_tb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/opt/anaconda3/envs/mywork/lib/python3.7/site-packages/keras/saving/legacy/serialization.py\u001b[0m in \u001b[0;36mdeserialize_keras_object\u001b[0;34m(identifier, module_objects, custom_objects, printable_module_name)\u001b[0m\n\u001b[1;32m    556\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mobj\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    557\u001b[0m                 raise ValueError(\n\u001b[0;32m--> 558\u001b[0;31m                     \u001b[0;34mf\"Unknown {printable_module_name}: '{object_name}'. \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    559\u001b[0m                     \u001b[0;34m\"Please ensure you are using a \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    560\u001b[0m                     \u001b[0;34m\"`keras.utils.custom_object_scope` \"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: Unknown metric function: 'f1score'. Please ensure you are using a `keras.utils.custom_object_scope` and that this object is included in the scope. See https://www.tensorflow.org/guide/keras/save_and_serialize#registering_the_custom_object for details."
     ]
    }
   ],
   "source": [
    "model = keras.models.load_model(\"my_cifar10_model.h5\")\n",
    "model.evaluate(X_valid, y_valid)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Reusing TensorBoard on port 6006 (pid 5286), started 0:45:25 ago. (Use '!kill 5286' to kill it.)"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\n",
       "      <iframe id=\"tensorboard-frame-783dc1bc16f65960\" width=\"100%\" height=\"800\" frameborder=\"0\">\n",
       "      </iframe>\n",
       "      <script>\n",
       "        (function() {\n",
       "          const frame = document.getElementById(\"tensorboard-frame-783dc1bc16f65960\");\n",
       "          const url = new URL(\"http://localhost\");\n",
       "          const port = 6006;\n",
       "          if (port) {\n",
       "            url.port = port;\n",
       "          }\n",
       "          frame.src = url;\n",
       "        })();\n",
       "      </script>\n",
       "    "
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%tensorboard --logdir=./my_cifar10_logs --port=6006"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "mywork",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.13"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "f85e5c70ea70364c03a66286146da5a7449ac12bc4658830fe6e69f9aca4feb9"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}