solutions/win2022/A3/fully_connected_networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "DDJwQPZcupab",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# EECS 498-007/598-005 Assignment 3-1: Fully-Connected Neural Networks and Dropout\n",
    "\n",
    "Before we start, please put your name and UMID in following format\n",
    "\n",
    ": Firstname LASTNAME, #00000000   //   e.g.) Justin JOHNSON, #12345678"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "tt7vq1h6mRto",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "**Your Answer:**   \n",
    "Your NAME, #XXXXXXXX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "FrfeHl_-m4V-",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Setup Code\n",
    "Before getting started, we need to run some boilerplate code to set up our environment, same as Assignment 1. You'll need to rerun this setup code each time you start the notebook.\n",
    "\n",
    "First, run this cell load the autoreload extension. This allows us to edit .py source files, and re-import them into the notebook for a seamless editing and debugging experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 845,
     "status": "ok",
     "timestamp": 1601234373609,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "VyQblYp0nEZq",
    "new_sheet": false,
    "outputId": "8b3b737f-6cc9-4ddc-ac19-4232065c2481",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "Q7ymI0aZ2W1b",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "### Google Colab Setup\n",
    "Next we need to run a few commands to set up our environment on Google Colab. If you are running this notebook on a local machine you can skip this section.\n",
    "\n",
    "Run the following cell to mount your Google Drive. Follow the link, sign in to your Google account (the same account you used to store this notebook!) and copy the authorization code into the text box that appears below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 1088,
     "status": "ok",
     "timestamp": 1601234373868,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "c_LLpLyC2eau",
    "new_sheet": false,
    "outputId": "57f679b3-5f31-471a-93e7-868e6b5019b7",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "mbq-UT8J2mnv",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now recall the path in your Google Drive where you uploaded this notebook, fill it in below. If everything is working correctly then running the folowing cell should print the filenames from the assignment:\n",
    "\n",
    "```\n",
    "['convolutional_networks.ipynb', 'fully_connected_networks.ipynb', 'eecs598', 'convolutional_networks.py', 'fully_connected_networks.py', 'a3_helper.py']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 54
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 1071,
     "status": "ok",
     "timestamp": 1601234373870,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "WcrhTOZW243H",
    "new_sheet": false,
    "outputId": "9af8c19e-96a4-40b8-c58b-f7eebaeccaa9",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# TODO: Fill in the Google Drive path where you uploaded the assignment\n",
    "# Example: If you create a 2022WI folder and put all the files under A3 folder, then '2022WI/A3'\n",
    "GOOGLE_DRIVE_PATH_AFTER_MYDRIVE = None\n",
    "GOOGLE_DRIVE_PATH = os.path.join('drive', 'My Drive', GOOGLE_DRIVE_PATH_AFTER_MYDRIVE)\n",
    "print(os.listdir(GOOGLE_DRIVE_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "xegb0uDA232J",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Once you have successfully mounted your Google Drive and located the path to this assignment, run th following cell to allow us to import from the `.py` files of this assignment. If it works correctly, it should print the message:\n",
    "\n",
    "```\n",
    "Hello from fully_connected_networks.py!\n",
    "Hello from a3_helper.py!\n",
    "```\n",
    "\n",
    "as well as the last edit time for the file `fully_connected_networks.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 1053,
     "status": "ok",
     "timestamp": 1601234373870,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "AhGQF5sw3Fas",
    "new_sheet": false,
    "outputId": "0f9e3f2d-d3d7-4ee9-d6d7-9b6ff7f816cd",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(GOOGLE_DRIVE_PATH)\n",
    "\n",
    "import time, os\n",
    "os.environ[\"TZ\"] = \"US/Eastern\"\n",
    "time.tzset()\n",
    "\n",
    "from fully_connected_networks import hello_fully_connected_networks\n",
    "hello_fully_connected_networks()\n",
    "\n",
    "from a3_helper import hello_helper\n",
    "hello_helper()\n",
    "\n",
    "fully_connected_networks_path = os.path.join(GOOGLE_DRIVE_PATH, 'fully_connected_networks.py')\n",
    "fully_connected_networks_edit_time = time.ctime(os.path.getmtime(fully_connected_networks_path))\n",
    "print('fully_connected_networks.py last edited on %s' % fully_connected_networks_edit_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ynKS05gJ4iBo",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "fN1SShPR4lJV",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Setup code\n",
    "Run some setup code for this notebook: Import some useful packages and increase the default figure size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "VUCKw4Tl1ddj",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "import torch\n",
    "import torchvision\n",
    "import matplotlib.pyplot as plt\n",
    "import statistics\n",
    "import random\n",
    "import time\n",
    "import math\n",
    "%matplotlib inline\n",
    "\n",
    "from eecs598 import reset_seed, Solver\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0)\n",
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "lhqpd2IN2O-K",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Starting in this assignment, we will use the GPU to accelerate our computation. Run this cell to make sure you are using a GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 1033,
     "status": "ok",
     "timestamp": 1601234373871,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "SGDxdBIMRX6b",
    "new_sheet": false,
    "outputId": "dacab859-f9cd-4419-85d1-7ac5464c7ec6",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "if torch.cuda.is_available:\n",
    "  print('Good to go!')\n",
    "else:\n",
    "  print('Please set GPU via Edit -> Notebook Settings.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "-Yv3zQYw5B3s",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Load the CIFAR-10 dataset\n",
    "Then, we will first load the CIFAR-10 dataset. The utility function `eecs598.data.preprocess_cifar10()` returns the entire CIFAR-10 dataset as a set of six **Torch tensors** while also preprocessing the RGB images:\n",
    "\n",
    "- `X_train` contains all training images (real numbers in the range $[0, 1]$)\n",
    "- `y_train` contains all training labels (integers in the range $[0, 9]$)\n",
    "- `X_val` contains all validation images\n",
    "- `y_val` contains all validation labels\n",
    "- `X_test` contains all test images\n",
    "- `y_test` contains all test labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 568
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5271,
     "status": "ok",
     "timestamp": 1601234378126,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "V2mFlFmQ1ddm",
    "new_sheet": false,
    "outputId": "6b1a14a3-421a-4bf2-e424-1ae15b343287",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Invoke the above function to get our data. \n",
    "import eecs598\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "data_dict = eecs598.data.preprocess_cifar10(cuda=True, dtype=torch.float64)\n",
    "print('Train data shape: ', data_dict['X_train'].shape)\n",
    "print('Train labels shape: ', data_dict['y_train'].shape)\n",
    "print('Validation data shape: ', data_dict['X_val'].shape)\n",
    "print('Validation labels shape: ', data_dict['y_val'].shape)\n",
    "print('Test data shape: ', data_dict['X_test'].shape)\n",
    "print('Test labels shape: ', data_dict['y_test'].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ZeH0OvuEe1CN",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Fully-connected neural networks\n",
    "In the previous homework you implemented a fully-connected two-layer neural network on CIFAR-10. The implementation was simple but not very modular since the loss and gradient were computed in a single monolithic function. This is manageable for a simple two-layer network, but would become impractical as we move to bigger models. Ideally we want to build networks using a more modular design so that we can implement different layer types in isolation and then snap them together into models with different architectures."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "3Qiu9_4pe1CP",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-ignore"
    ]
   },
   "source": [
    "In this exercise we will implement fully-connected networks using a more modular approach. For each layer we will implement a `forward` and a `backward` function. The `forward` function will receive inputs, weights, and other parameters and will return both an output and a `cache` object storing data needed for the backward pass, like this:\n",
    "\n",
    "```python\n",
    "def forward(x, w):\n",
    "  \"\"\" Receive inputs x and weights w \"\"\"\n",
    "  # Do some computations ...\n",
    "  z = # ... some intermediate value\n",
    "  # Do some more computations ...\n",
    "  out = # the output\n",
    "   \n",
    "  cache = (x, w, z, out) # Values we need to compute gradients\n",
    "   \n",
    "  return out, cache\n",
    "```\n",
    "\n",
    "The backward pass will receive upstream derivatives and the `cache` object, and will return gradients with respect to the inputs and weights, like this:\n",
    "\n",
    "```python\n",
    "def backward(dout, cache):\n",
    "  \"\"\"\n",
    "  Receive dout (derivative of loss with respect to outputs) and cache,\n",
    "  and compute derivative with respect to inputs.\n",
    "  \"\"\"\n",
    "  # Unpack cache values\n",
    "  x, w, z, out = cache\n",
    "  \n",
    "  # Use values in cache to compute derivatives\n",
    "  dx = # Derivative of loss with respect to x\n",
    "  dw = # Derivative of loss with respect to w\n",
    "  \n",
    "  return dx, dw\n",
    "```\n",
    "\n",
    "After implementing a bunch of layers this way, we will be able to easily combine them to build classifiers with different architectures.\n",
    "\n",
    "In addition to implementing fully-connected networks of arbitrary depth, we will also explore different update rules for optimization, and introduce Dropout as a regularizer as a tool to more efficiently optimize deep networks.\n",
    "\n",
    "To validate our implementation, we will compare the analytically computed gradients with numerical approximations of the gradient as done in previous assignments. You can inspect the numeric gradient function `compute_numeric_gradient` in `eecs598/grad.py`. Please note that we have updated the function to accept upstream gradients to allow us to debug intermediate layers easily. You can check the update description by running the code block below.\n",
    "  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 442
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5256,
     "status": "ok",
     "timestamp": 1601234378127,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "DfseUp7H-TF_",
    "new_sheet": false,
    "outputId": "2c05fb93-f8c1-48cf-8a80-2dbfb27130bb",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "help(eecs598.grad.compute_numeric_gradient)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "JB7Eu3qJ9xnm",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Linear layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "bRdnxsvZunFu",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "For each layer we implement, we will define a class with two static methods `forward` and `backward`. The class structure is currently provided in `fully_connected_layers.py`, you will be implementing both the `forward` and `backward` methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "0NNv3l-ne1Cb",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Linear layer: forward\n",
    "Implement the `Linear.forward` function in `fully_connected_layers.py`. Once you are done you can test your implementaion by running the next cell. You should see errors less than `1e-7`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5239,
     "status": "ok",
     "timestamp": 1601234378127,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "sjq2Sq4Ze1Cc",
    "new_sheet": false,
    "outputId": "4a32ddfb-49aa-4d56-b73b-89b96c67479a",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import Linear\n",
    "\n",
    "# Test the Linear.forward function\n",
    "num_inputs = 2\n",
    "input_shape = torch.tensor((4, 5, 6))\n",
    "output_dim = 3\n",
    "\n",
    "input_size = num_inputs * torch.prod(input_shape)\n",
    "weight_size = output_dim * torch.prod(input_shape)\n",
    "\n",
    "x = torch.linspace(-0.1, 0.5, steps=input_size, dtype=torch.float64, device='cuda')\n",
    "w = torch.linspace(-0.2, 0.3, steps=weight_size, dtype=torch.float64, device='cuda')\n",
    "b = torch.linspace(-0.3, 0.1, steps=output_dim, dtype=torch.float64, device='cuda')\n",
    "x = x.reshape(num_inputs, *input_shape)\n",
    "w = w.reshape(torch.prod(input_shape), output_dim)\n",
    "\n",
    "out, _ = Linear.forward(x, w, b)\n",
    "correct_out = torch.tensor([[1.49834984, 1.70660150, 1.91485316],\n",
    "                            [3.25553226, 3.51413301, 3.77273372]]\n",
    "                            ).double().cuda()\n",
    "\n",
    "print('Testing Linear.forward function:')\n",
    "print('difference: ', eecs598.grad.rel_error(out, correct_out))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "4mxIDo46e1Cf",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Linear layer: backward\n",
    "Now implement the `Linear.backward` function and test your implementation using numeric gradient checking. \n",
    "\n",
    "Run the following to test your implementation of `Linear.backward`. You should see errors less than `1e-7`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5223,
     "status": "ok",
     "timestamp": 1601234378128,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "ts85gmote1Cg",
    "new_sheet": false,
    "outputId": "1bc280a6-9ef1-4364-d072-8d962f81832c",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import Linear\n",
    "\n",
    "# Test the Linear.backward function\n",
    "reset_seed(0)\n",
    "x = torch.randn(10, 2, 3, dtype=torch.float64, device='cuda')\n",
    "w = torch.randn(6, 5, dtype=torch.float64, device='cuda')\n",
    "b = torch.randn(5, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(10, 5, dtype=torch.float64, device='cuda')\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda x: Linear.forward(x, w, b)[0], x, dout)\n",
    "dw_num = eecs598.grad.compute_numeric_gradient(lambda w: Linear.forward(x, w, b)[0], w, dout)\n",
    "db_num = eecs598.grad.compute_numeric_gradient(lambda b: Linear.forward(x, w, b)[0], b, dout)\n",
    "\n",
    "_, cache = Linear.forward(x, w, b)\n",
    "dx, dw, db = Linear.backward(dout, cache)\n",
    "\n",
    "# The error should be around e-10 or less\n",
    "print('Testing Linear.backward function:')\n",
    "print('dx error: ', eecs598.grad.rel_error(dx_num, dx))\n",
    "print('dw error: ', eecs598.grad.rel_error(dw_num, dw))\n",
    "print('db error: ', eecs598.grad.rel_error(db_num, db))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "bdIqQzqiJQE6",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# ReLU activation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "YdX98A_qvTRt",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "We will now implement the ReLU nonlinearity. As above, we will define a class with two empty static methods, and implement them in upcoming cells. The class structure can be found in `fully_connected_networks.py`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "n2DyqL4Ae1Cl",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## ReLU activation: forward\n",
    "Implement the forward pass for the ReLU activation function in the `ReLU.forward` function. You **should not** change the input tensor with an in-place operation. \n",
    "\n",
    "Run the following to test your implementation of the ReLU forward pass. Your errors should be less than `1e-7`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5207,
     "status": "ok",
     "timestamp": 1601234378129,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "QblpieUJe1Cm",
    "new_sheet": false,
    "outputId": "3e7cf951-3c8f-44eb-b5da-ba72042d5184",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import ReLU\n",
    "\n",
    "reset_seed(0)\n",
    "x = torch.linspace(-0.5, 0.5, steps=12, dtype=torch.float64, device='cuda')\n",
    "x = x.reshape(3, 4)\n",
    "\n",
    "out, _ = ReLU.forward(x)\n",
    "correct_out = torch.tensor([[ 0.,          0.,          0.,          0.,        ],\n",
    "                            [ 0.,          0.,          0.04545455,  0.13636364,],\n",
    "                            [ 0.22727273,  0.31818182,  0.40909091,  0.5,       ]],\n",
    "                            dtype=torch.float64,\n",
    "                            device='cuda')\n",
    "\n",
    "# Compare your output with ours. The error should be on the order of e-8\n",
    "print('Testing ReLU.forward function:')\n",
    "print('difference: ', eecs598.grad.rel_error(out, correct_out))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "3bSInb7xe1Cq",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## ReLU activation: backward\n",
    "Now implement the backward pass for the ReLU activation function.\n",
    "\n",
    "Again, you should not change the input tensor with an in-place operation.\n",
    "\n",
    "Run the following to test your implementation of `ReLU.backward`. Your errors should be less than `1e-8`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5190,
     "status": "ok",
     "timestamp": 1601234378129,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "odiV48zBe1Cr",
    "new_sheet": false,
    "outputId": "710d6e24-1422-4684-92f2-b40f3f68cbd7",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import ReLU\n",
    "\n",
    "reset_seed(0)\n",
    "x = torch.randn(10, 10, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(*x.shape, dtype=torch.float64, device='cuda')\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda x: ReLU.forward(x)[0], x, dout)\n",
    "\n",
    "_, cache = ReLU.forward(x)\n",
    "dx = ReLU.backward(dout, cache)\n",
    "\n",
    "# The error should be on the order of e-12\n",
    "print('Testing ReLU.backward function:')\n",
    "print('dx error: ', eecs598.grad.rel_error(dx_num, dx))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "eVTMuUOZe1Cv",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# \"Sandwich\" layers\n",
    "There are some common patterns of layers that are frequently used in neural nets. For example, linear layers are frequently followed by a ReLU nonlinearity. To make these common patterns easy, we define a convenience layer.\n",
    "\n",
    "This also shows how our layer abstraction allows us to implement new layers by composing existing layer implementations. This is a powerful mechanism for structuring deep learning code in a modular fashion.\n",
    "\n",
    "For now take a look at the `forward` and `backward` functions in `Linear_ReLU`, and run the following to numerically gradient check the backward pass.\n",
    "\n",
    "Run the following to test the implementation of the `Linear_ReLU` layer using numeric gradient checking. You should see errors less than `1e-8`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5173,
     "status": "ok",
     "timestamp": 1601234378129,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "-gaY5YfAe1Cw",
    "new_sheet": false,
    "outputId": "7370db9d-80f6-4c3d-9976-b67ad6a2ceff",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import Linear_ReLU\n",
    "\n",
    "reset_seed(0)\n",
    "x = torch.randn(2, 3, 4, dtype=torch.float64, device='cuda')\n",
    "w = torch.randn(12, 10, dtype=torch.float64, device='cuda')\n",
    "b = torch.randn(10, dtype=torch.float64, device='cuda')\n",
    "dout = torch.randn(2, 10, dtype=torch.float64, device='cuda')\n",
    "\n",
    "out, cache = Linear_ReLU.forward(x, w, b)\n",
    "dx, dw, db = Linear_ReLU.backward(dout, cache)\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda x: Linear_ReLU.forward(x, w, b)[0], x, dout)\n",
    "dw_num = eecs598.grad.compute_numeric_gradient(lambda w: Linear_ReLU.forward(x, w, b)[0], w, dout)\n",
    "db_num = eecs598.grad.compute_numeric_gradient(lambda b: Linear_ReLU.forward(x, w, b)[0], b, dout)\n",
    "\n",
    "# Relative error should be around e-8 or less\n",
    "print('Testing Linear_ReLU.forward and Linear_ReLU.backward:')\n",
    "print('dx error: ', eecs598.grad.rel_error(dx_num, dx))\n",
    "print('dw error: ', eecs598.grad.rel_error(dw_num, dw))\n",
    "print('db error: ', eecs598.grad.rel_error(db_num, db))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "rAGgiyP5e1C0",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Loss layers: Softmax and SVM\n",
    "You implemented these loss functions in the last assignment, so we'll give them to you for free in `helper_functions.py`. You should still make sure you understand how they work by looking at the implementations. We can first verify our implementations.\n",
    "\n",
    "\n",
    "Run the following to perform numeric gradient checking on the two loss functions. You should see errors less than `1e-7` for svm_loss and `1e-6` for softmax_loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 136
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 5155,
     "status": "ok",
     "timestamp": 1601234378130,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "BU9xp64De1C1",
    "new_sheet": false,
    "outputId": "ea85ba8b-be43-404a-91a9-5037fb3b736b",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from a3_helper import svm_loss, softmax_loss\n",
    "\n",
    "reset_seed(0)\n",
    "num_classes, num_inputs = 10, 50\n",
    "x = 0.001 * torch.randn(num_inputs, num_classes, dtype=torch.float64, device='cuda')\n",
    "y = torch.randint(num_classes, size=(num_inputs,), dtype=torch.int64, device='cuda')\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda x: svm_loss(x, y)[0], x)\n",
    "loss, dx = svm_loss(x, y)\n",
    "\n",
    "# Test svm_loss function. Loss should be around 9 and dx error should be around the order of e-9\n",
    "print('Testing svm_loss:')\n",
    "print('loss: ', loss.item())\n",
    "print('dx error: ', eecs598.grad.rel_error(dx_num, dx))\n",
    "\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda x: softmax_loss(x, y)[0], x)\n",
    "loss, dx = softmax_loss(x, y)\n",
    "\n",
    "# Test softmax_loss function. Loss should be close to 2.3 and dx error should be around e-8\n",
    "print('\\nTesting softmax_loss:')\n",
    "print('loss: ', loss.item())\n",
    "print('dx error: ', eecs598.grad.rel_error(dx_num, dx))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "qq7-cyfQe1C4",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Two-layer network\n",
    "In the previous assignment you implemented a two-layer neural network in a single monolithic class. Now that you have implemented modular versions of the necessary layers, you will reimplement the two layer network using these modular implementations.\n",
    "\n",
    "Complete the implementation of the `TwoLayerNet` class. This class will serve as a model for the other networks you will implement in this assignment, so read through it to make sure you understand the API. \n",
    "\n",
    "Once you have finished implementing the forward and backward passes of your two-layer net, run the following to test your implementation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 238
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 7734,
     "status": "ok",
     "timestamp": 1601234380728,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "d3JOcfyze1C5",
    "new_sheet": false,
    "outputId": "2d38bb9e-f8fe-4a58-dc79-f817bab4949e",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import TwoLayerNet\n",
    "from a3_helper import svm_loss, softmax_loss\n",
    "\n",
    "reset_seed(0)\n",
    "N, D, H, C = 3, 5, 50, 7\n",
    "X = torch.randn(N, D, dtype=torch.float64, device='cuda')\n",
    "y = torch.randint(C, size=(N,), dtype=torch.int64, device='cuda')\n",
    "\n",
    "std = 1e-3\n",
    "model = TwoLayerNet(\n",
    "          input_dim=D,\n",
    "          hidden_dim=H,\n",
    "          num_classes=C,\n",
    "          weight_scale=std,\n",
    "          dtype=torch.float64,\n",
    "          device='cuda'\n",
    "        )\n",
    "\n",
    "print('Testing initialization ... ')\n",
    "W1_std = torch.abs(model.params['W1'].std() - std)\n",
    "b1 = model.params['b1']\n",
    "W2_std = torch.abs(model.params['W2'].std() - std)\n",
    "b2 = model.params['b2']\n",
    "assert W1_std < std / 10, 'First layer weights do not seem right'\n",
    "assert torch.all(b1 == 0), 'First layer biases do not seem right'\n",
    "assert W2_std < std / 10, 'Second layer weights do not seem right'\n",
    "assert torch.all(b2 == 0), 'Second layer biases do not seem right'\n",
    "\n",
    "print('Testing test-time forward pass ... ')\n",
    "model.params['W1'] = torch.linspace(-0.7, 0.3, steps=D * H, dtype=torch.float64, device='cuda').reshape(D, H)\n",
    "model.params['b1'] = torch.linspace(-0.1, 0.9, steps=H, dtype=torch.float64, device='cuda')\n",
    "model.params['W2'] = torch.linspace(-0.3, 0.4, steps=H * C, dtype=torch.float64, device='cuda').reshape(H, C)\n",
    "model.params['b2'] = torch.linspace(-0.9, 0.1, steps=C, dtype=torch.float64, device='cuda')\n",
    "X = torch.linspace(-5.5, 4.5, steps=N * D, dtype=torch.float64, device='cuda').reshape(D, N).t()\n",
    "scores = model.loss(X)\n",
    "correct_scores = torch.tensor(\n",
    "  [[11.53165108,  12.2917344,   13.05181771,  13.81190102,  14.57198434, 15.33206765,  16.09215096],\n",
    "   [12.05769098,  12.74614105,  13.43459113,  14.1230412,   14.81149128, 15.49994135,  16.18839143],\n",
    "   [12.58373087,  13.20054771,  13.81736455,  14.43418138,  15.05099822, 15.66781506,  16.2846319 ]],\n",
    "    dtype=torch.float64, device='cuda')\n",
    "scores_diff = torch.abs(scores - correct_scores).sum()\n",
    "assert scores_diff < 1e-6, 'Problem with test-time forward pass'\n",
    "\n",
    "print('Testing training loss (no regularization)')\n",
    "y = torch.tensor([0, 5, 1])\n",
    "loss, grads = model.loss(X, y)\n",
    "correct_loss = 3.4702243556\n",
    "assert abs(loss - correct_loss) < 1e-10, 'Problem with training-time loss'\n",
    "\n",
    "model.reg = 1.0\n",
    "loss, grads = model.loss(X, y)\n",
    "correct_loss = 49.719461034881775\n",
    "assert abs(loss - correct_loss) < 1e-10, 'Problem with regularization loss'\n",
    "\n",
    "# Errors should be around e-6 or less\n",
    "for reg in [0.0, 0.7]:\n",
    "  print('Running numeric gradient check with reg = ', reg)\n",
    "  model.reg = reg\n",
    "  loss, grads = model.loss(X, y)\n",
    "\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eecs598.grad.compute_numeric_gradient(f, model.params[name])\n",
    "    print('%s relative error: %.2e' % (name, eecs598.grad.rel_error(grad_num, grads[name])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "q1Odj9XQe1C9",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Solver\n",
    "In the previous assignment, the logic for training models was coupled to the models themselves. Following a more modular design, for this assignment we have split the logic for training models into a separate class.\n",
    "\n",
    "Read through `help(Solver)` to familiarize yourself with the API. After doing so, use a `Solver` instance to train a `TwoLayerNet` that achieves at least `50%` accuracy on the validation set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 7716,
     "status": "ok",
     "timestamp": 1601234380729,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "lZ-8wKffRoDu",
    "new_sheet": false,
    "outputId": "5dc23ff2-c195-4307-edd3-f63265d2f5e5",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "print(help(Solver))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "ldQOcCK46YYx",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Use the Solver classe to create a solver instance that trains a TwoLayerNet to achieve at least 50% performance on the validation set. \n",
    "\n",
    "**Implement** `create_solver_instance` in `fully_connected_networks.py` to return a solver instance. Make sure to initialize the Solver instance with the input device."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 884
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 28150,
     "status": "ok",
     "timestamp": 1601234401182,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "6unJrOule1C_",
    "new_sheet": false,
    "outputId": "ca6ccbf0-25a9-4545-e07c-5ee919c9a7e5",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import create_solver_instance\n",
    "\n",
    "reset_seed(0)\n",
    "\n",
    "# Create a solver instance that achieves 50% performance on the validation set\n",
    "solver = create_solver_instance(data_dict=data_dict, dtype=torch.float64, device='cuda')\n",
    "solver.train()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 745
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 28590,
     "status": "ok",
     "timestamp": 1601234401636,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "gSSy7LTde1DE",
    "new_sheet": false,
    "outputId": "c9eba350-22f4-41c1-caa8-93383663c870",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Run this cell to visualize training loss and train / val accuracy\n",
    "plt.subplot(2, 1, 1)\n",
    "plt.title('Training loss')\n",
    "plt.plot(solver.loss_history, 'o')\n",
    "plt.xlabel('Iteration')\n",
    "\n",
    "plt.subplot(2, 1, 2)\n",
    "plt.title('Accuracy')\n",
    "plt.plot(solver.train_acc_history, '-o', label='train')\n",
    "plt.plot(solver.val_acc_history, '-o', label='val')\n",
    "plt.plot([0.5] * len(solver.val_acc_history), 'k--')\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend(loc='lower right')\n",
    "plt.gcf().set_size_inches(15, 12)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "oUwvMomE31Mh",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "If you're happy with the model's perfromance, run the following cell to save it. \n",
    "\n",
    "We will also reload the model and run it on validation to verify it's the right weights."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 28847,
     "status": "ok",
     "timestamp": 1601234401902,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "AfE_2VVK31fa",
    "new_sheet": false,
    "outputId": "20e20bf5-837b-4851-ed92-ef9fe0160eeb",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "path = os.path.join(GOOGLE_DRIVE_PATH, 'best_two_layer_net.pth')\n",
    "solver.model.save(path)\n",
    "\n",
    "# Create a new instance\n",
    "from fully_connected_networks import create_solver_instance\n",
    "reset_seed(0)\n",
    "\n",
    "solver = create_solver_instance(data_dict=data_dict, dtype=torch.float64, device='cuda')\n",
    "\n",
    "# Load model\n",
    "solver.model.load(path, dtype=torch.float64, device='cuda')\n",
    "\n",
    "# Evaluate on validation set\n",
    "accuracy = solver.check_accuracy(solver.X_val, solver.y_val)\n",
    "print(f\"Saved model's accuracy on validation is {accuracy}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "eNyFLT1We1DI",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Multilayer network\n",
    "Next you will implement a fully-connected network with an arbitrary number of hidden layers.\n",
    "\n",
    "Read through the `FullyConnectedNet` class in `fully_connected_networks.py`. Implement the initialization, the forward pass, and the backward pass. For the moment don't worry about implementing dropout; we will add this feature soon."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "3abR1_qhe1DK",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Initial loss and gradient check\n",
    "\n",
    "As a sanity check, run the following to check the initial loss and to gradient check the network both with and without regularization. Do the initial losses seem reasonable?\n",
    "\n",
    "For gradient checking, you should expect to see errors less than `1e-6`, except for the check on `W1` and `W2` with `reg=0` where your errors should be less than `1e-5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 34953,
     "status": "ok",
     "timestamp": 1601234408023,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "1waPtKRDe1DL",
    "new_sheet": false,
    "outputId": "b55b6750-f048-4e5f-97c1-d5b09610b385",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import FullyConnectedNet\n",
    "\n",
    "reset_seed(0)\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = torch.randn(N, D, dtype=torch.float64, device='cuda')\n",
    "y = torch.randint(C, size=(N,), dtype=torch.int64, device='cuda')\n",
    "\n",
    "for reg in [0, 3.14]:\n",
    "  print('Running check with reg = ', reg)\n",
    "  model = FullyConnectedNet(\n",
    "        [H1, H2], \n",
    "        input_dim=D,\n",
    "        num_classes=C,\n",
    "        reg=reg,\n",
    "        weight_scale=5e-2, \n",
    "        dtype=torch.float64, \n",
    "        device='cuda'\n",
    "  )\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss.item())\n",
    "\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eecs598.grad.compute_numeric_gradient(f, model.params[name])\n",
    "    print('%s relative error: %.2e' % (name, eecs598.grad.rel_error(grad_num, grads[name])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "-q6aWzNfe1DQ",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "As another sanity check, make sure you can overfit a small dataset of 50 images. First we will try a three-layer network with 100 units in each hidden layer. In the following cell, tweak the **learning rate** and **weight initialization scale** to overfit and achieve 100% training accuracy within 20 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 953
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 35371,
     "status": "ok",
     "timestamp": 1601234408445,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "2NccCDJ3e1DR",
    "new_sheet": false,
    "outputId": "f1fbf7a9-ecdc-497e-eaab-9f3d40c8b7fc",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import FullyConnectedNet, get_three_layer_network_params\n",
    "\n",
    "# TODO: Use a three-layer Net to overfit 50 training examples by \n",
    "# tweaking just the learning rate and initialization scale.\n",
    "reset_seed(0)\n",
    "\n",
    "num_train = 50\n",
    "small_data = {\n",
    "  'X_train': data_dict['X_train'][:num_train],\n",
    "  'y_train': data_dict['y_train'][:num_train],\n",
    "  'X_val': data_dict['X_val'],\n",
    "  'y_val': data_dict['y_val'],\n",
    "}\n",
    "\n",
    "# Update parameters in get_three_layer_network_params\n",
    "weight_scale, learning_rate = get_three_layer_network_params()\n",
    "\n",
    "model = FullyConnectedNet([100, 100],\n",
    "              weight_scale=weight_scale, dtype=torch.float32, device='cuda')\n",
    "solver = Solver(model, small_data,\n",
    "                print_every=10, num_epochs=20, batch_size=25,\n",
    "                optim_config={\n",
    "                  'learning_rate': learning_rate,\n",
    "                },\n",
    "                device='cuda',\n",
    "         )\n",
    "solver.train()\n",
    "\n",
    "plt.plot(solver.loss_history, 'o')\n",
    "plt.title('Training loss history')\n",
    "plt.xlabel('Iteration')\n",
    "plt.ylabel('Training loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "tskjw8VUe1DV",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Now try to use a five-layer network with 100 units on each layer to overfit 50 training examples. Again, you will have to adjust the learning rate and weight initialization scale, but you should be able to achieve 100% training accuracy within 20 epochs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 953
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 36411,
     "status": "ok",
     "timestamp": 1601234409494,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "D5mAWrrPe1Dc",
    "new_sheet": false,
    "outputId": "70c9ed1c-4214-4114-8c3e-593e8b8de64e",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import FullyConnectedNet, get_five_layer_network_params\n",
    "\n",
    "# TODO: Use a five-layer Net to overfit 50 training examples by \n",
    "# tweaking just the learning rate and initialization scale.\n",
    "reset_seed(0)\n",
    "\n",
    "num_train = 50\n",
    "small_data = {\n",
    "  'X_train': data_dict['X_train'][:num_train],\n",
    "  'y_train': data_dict['y_train'][:num_train],\n",
    "  'X_val': data_dict['X_val'],\n",
    "  'y_val': data_dict['y_val'],\n",
    "}\n",
    "\n",
    "\n",
    "# Update parameters in get_three_layer_network_params\n",
    "weight_scale, learning_rate = get_five_layer_network_params()\n",
    "\n",
    "# Run models and solver with parameters\n",
    "model = FullyConnectedNet([100, 100, 100, 100],\n",
    "                weight_scale=weight_scale, dtype=torch.float32, device='cuda')\n",
    "solver = Solver(model, small_data,\n",
    "                print_every=10, num_epochs=20, batch_size=25,\n",
    "                optim_config={\n",
    "                  'learning_rate': learning_rate,\n",
    "                },\n",
    "                device='cuda',\n",
    "         )\n",
    "# Turn off keep_best_params to allow final weights to be saved, instead of best weights on validation set.\n",
    "solver.train(return_best_params=False)\n",
    "\n",
    "plt.plot(solver.loss_history, 'o')\n",
    "plt.title('Training loss history')\n",
    "plt.xlabel('Iteration')\n",
    "plt.ylabel('Training loss')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "3M2JQjj_93RW",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "If you're satisfied with your model's performance, save the overfit model. Just a sanity check, we evaluate it one the training set again to verify that the saved weights have the correct performance. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 36393,
     "status": "ok",
     "timestamp": 1601234409495,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "YAnM8V9z938Q",
    "new_sheet": false,
    "outputId": "599233ed-1b62-47a3-c7f8-f6a945eb6607",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Set path \n",
    "path = os.path.join(GOOGLE_DRIVE_PATH, 'best_overfit_five_layer_net.pth')\n",
    "solver.model.save(path)\n",
    "\n",
    "\n",
    "# Create a new instance  -- Note that hidden dims being different doesn't matter here.\n",
    "model = FullyConnectedNet(hidden_dims=[100, ], dtype=torch.float32, device='cuda')\n",
    "solver = Solver(model, small_data,\n",
    "                print_every=10, num_epochs=20, batch_size=25,\n",
    "                optim_config={\n",
    "                  'learning_rate': learning_rate,\n",
    "                },\n",
    "                device='cuda',\n",
    "         )\n",
    "\n",
    "\n",
    "# Load model\n",
    "solver.model.load(path, dtype=torch.float32, device='cuda')\n",
    "\n",
    "# Evaluate on validation set\n",
    "accuracy = solver.check_accuracy(solver.X_train, solver.y_train)\n",
    "print(f\"Saved model's accuracy on small train is {accuracy}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "T4eWrnY7e1Di",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Update rules\n",
    "So far we have used vanilla stochastic gradient descent (SGD) as our update rule. More sophisticated update rules can make it easier to train deep networks. We will implement a few of the most commonly used update rules and compare them to vanilla SGD."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "zBDJqbeVe1Dn",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## SGD+Momentum\n",
    "Stochastic gradient descent with momentum is a widely used update rule that tends to make deep networks converge faster than vanilla stochastic gradient descent. See the Momentum Update section at http://cs231n.github.io/neural-networks-3/#sgd for more information.\n",
    "\n",
    "We will implement various first-order update rules that are commonly used\n",
    "for training neural networks. Each update rule accepts current weights and the\n",
    "gradient of the loss with respect to those weights and produces the next set of\n",
    "weights. Each update rule has the same interface:\n",
    "```python\n",
    "def update(w, dw, config=None):\n",
    "Inputs:\n",
    "  - w: A tensor giving the current weights.\n",
    "  - dw: A tensor of the same shape as w giving the gradient of the\n",
    "    loss with respect to w.\n",
    "  - config: A dictionary containing hyperparameter values such as learning\n",
    "    rate, momentum, etc. If the update rule requires caching values over many\n",
    "    iterations, then config will also hold these cached values.\n",
    "Returns:\n",
    "  - next_w: The next point after the update.\n",
    "  - config: The config dictionary to be passed to the next iteration of the\n",
    "    update rule.\n",
    "NOTE: For most update rules, the default learning rate will probably not\n",
    "perform well; however the default values of the other hyperparameters should\n",
    "work well for a variety of different problems.\n",
    "For efficiency, update rules may perform in-place updates, mutating w and\n",
    "setting next_w equal to w.\n",
    "```\n",
    "We provide the implementation of the SGD update rule for your reference in `fully_connected_networks.py`\n",
    "\n",
    "Now **implement** the SGD+Momentum update rule using the same interface. Run the following to check your implementation of SGD+Momentum. You should see errors less than `1e-7`.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 36378,
     "status": "ok",
     "timestamp": 1601234409495,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "RbQrkNo_e1Dp",
    "new_sheet": false,
    "outputId": "aeccd6f4-eb31-461a-9fe9-5806e912f6fd",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import sgd_momentum\n",
    "\n",
    "reset_seed(0)\n",
    "\n",
    "N, D = 4, 5\n",
    "w = torch.linspace(-0.4, 0.6, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "dw = torch.linspace(-0.6, 0.4, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "v = torch.linspace(0.6, 0.9, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "\n",
    "config = {'learning_rate': 1e-3, 'velocity': v}\n",
    "next_w, _ = sgd_momentum(w, dw, config=config)\n",
    "\n",
    "expected_next_w = torch.tensor([\n",
    "  [ 0.1406,      0.20738947,  0.27417895,  0.34096842,  0.40775789],\n",
    "  [ 0.47454737,  0.54133684,  0.60812632,  0.67491579,  0.74170526],\n",
    "  [ 0.80849474,  0.87528421,  0.94207368,  1.00886316,  1.07565263],\n",
    "  [ 1.14244211,  1.20923158,  1.27602105,  1.34281053,  1.4096    ]],\n",
    "   dtype=torch.float64, device='cuda')\n",
    "expected_velocity = torch.tensor([\n",
    "  [ 0.5406,      0.55475789,  0.56891579, 0.58307368,  0.59723158],\n",
    "  [ 0.61138947,  0.62554737,  0.63970526,  0.65386316,  0.66802105],\n",
    "  [ 0.68217895,  0.69633684,  0.71049474,  0.72465263,  0.73881053],\n",
    "  [ 0.75296842,  0.76712632,  0.78128421,  0.79544211,  0.8096    ]],\n",
    "   dtype=torch.float64, device='cuda')\n",
    "\n",
    "# Should see relative errors around e-8 or less\n",
    "print('next_w error: ', eecs598.grad.rel_error(next_w, expected_next_w))\n",
    "print('velocity error: ', eecs598.grad.rel_error(expected_velocity, config['velocity']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "7QQj73zje1D2",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Once you have done so, run the following to train a six-layer network with both SGD and SGD+momentum. You should see the SGD+momentum update rule converge faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 38894,
     "status": "ok",
     "timestamp": 1601234412015,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "qXdMNC9Ve1D4",
    "new_sheet": false,
    "outputId": "94f7c217-595c-4f25-dbd1-594257766175",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import FullyConnectedNet, sgd, sgd_momentum\n",
    "\n",
    "# TODO: Use a three-layer Net to overfit 50 training examples by \n",
    "# tweaking just the learning rate and initialization scale.\n",
    "reset_seed(0)\n",
    "\n",
    "num_train = 4000\n",
    "small_data = {\n",
    "  'X_train': data_dict['X_train'][:num_train],\n",
    "  'y_train': data_dict['y_train'][:num_train],\n",
    "  'X_val': data_dict['X_val'],\n",
    "  'y_val': data_dict['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "\n",
    "for update_rule_name, update_rule_fn in [('sgd', sgd), ('sgd_momentum', sgd_momentum)]:\n",
    "  print('running with ', update_rule_name)\n",
    "  model = FullyConnectedNet([100, 100, 100, 100, 100], weight_scale=5e-2,\n",
    "                            dtype=torch.float32, device='cuda')\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=5, batch_size=100,\n",
    "                  update_rule=update_rule_fn,\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-2,\n",
    "                  },\n",
    "                  print_every=1000,\n",
    "                  verbose=True,\n",
    "                  device='cuda')\n",
    "  solvers[update_rule_name] = solver\n",
    "  solver.train()\n",
    "  print()\n",
    "  \n",
    "plt.subplot(3, 1, 1)\n",
    "plt.title('Training loss')\n",
    "plt.xlabel('Iteration')\n",
    "for update_rule, solver in solvers.items():\n",
    "  plt.plot(solver.loss_history, 'o', label=\"loss_%s\" % update_rule)\n",
    "plt.legend(loc='lower center', ncol=4)\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "plt.title('Training accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "for update_rule, solver in solvers.items():\n",
    "  plt.plot(solver.train_acc_history, '-o', label=\"train_acc_%s\" % update_rule)\n",
    "plt.legend(loc='lower center', ncol=4)\n",
    "\n",
    "  \n",
    "plt.subplot(3, 1, 3)\n",
    "plt.title('Validation accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "for update_rule, solver in solvers.items():\n",
    "  plt.plot(solver.val_acc_history, '-o', label=\"val_acc_%s\" % update_rule)\n",
    "plt.legend(loc='lower center', ncol=4)\n",
    "\n",
    "plt.gcf().set_size_inches(10, 20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "wYtKqDdEe1D-",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## RMSProp\n",
    "RMSProp [1] is an update rule that set per-parameter learning rates by using a running average of the second moments of gradients.\n",
    "\n",
    "**Implement** the RMSProp update rule in the `rmsprop` function in `fully_connected_networks.py`. Run the following to test your RMSProp implementation. You should see errors less than `1e-6`.\n",
    "\n",
    "[1] Tijmen Tieleman and Geoffrey Hinton. \"Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude.\" COURSERA: Neural Networks for Machine Learning 4 (2012)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 38890,
     "status": "ok",
     "timestamp": 1601234412016,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "RBBpJhJie1D_",
    "new_sheet": false,
    "outputId": "36a80bcb-2e97-45e5-a516-34df6eed7749",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Test RMSProp implementation\n",
    "from fully_connected_networks import rmsprop\n",
    "\n",
    "reset_seed(0)\n",
    "\n",
    "N, D = 4, 5\n",
    "w = torch.linspace(-0.4, 0.6, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "dw = torch.linspace(-0.6, 0.4, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "cache = torch.linspace(0.6, 0.9, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "\n",
    "config = {'learning_rate': 1e-2, 'cache': cache}\n",
    "next_w, _ = rmsprop(w, dw, config=config)\n",
    "\n",
    "expected_next_w = torch.tensor([\n",
    "  [-0.39223849, -0.34037513, -0.28849239, -0.23659121, -0.18467247],\n",
    "  [-0.132737,   -0.08078555, -0.02881884,  0.02316247,  0.07515774],\n",
    "  [ 0.12716641,  0.17918792,  0.23122175,  0.28326742,  0.33532447],\n",
    "  [ 0.38739248,  0.43947102,  0.49155973,  0.54365823,  0.59576619]],\n",
    "   dtype=torch.float64, device='cuda')\n",
    "expected_cache = torch.tensor([\n",
    "  [ 0.5976,      0.6126277,   0.6277108,   0.64284931,  0.65804321],\n",
    "  [ 0.67329252,  0.68859723,  0.70395734,  0.71937285,  0.73484377],\n",
    "  [ 0.75037008,  0.7659518,   0.78158892,  0.79728144,  0.81302936],\n",
    "  [ 0.82883269,  0.84469141,  0.86060554,  0.87657507,  0.8926    ]],\n",
    "   dtype=torch.float64, device='cuda')\n",
    "\n",
    "print('next_w error: ', eecs598.grad.rel_error(expected_next_w, next_w))\n",
    "print('cache error: ', eecs598.grad.rel_error(expected_cache, config['cache']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "bMdq7WRFDiJw",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Adam\n",
    "Adam [2] extends RMSprop with a first-order gradient cache similar to momentum, and a bias correction mechanism to prevent large steps at the start of optimization. Adam is one of the most commonly used update rules used in practice for training deep neural networks.\n",
    "\n",
    "Implement the Adam update rule in the `adam` function in `fully_connected_networks.py`. Run the following to test your Adam implementation. You should see error less than `1e-6` for `next_w`, and errors less than `1e-8` for `v` and `m`:\n",
    "\n",
    "\n",
    "\n",
    "**NOTE:** Please implement the _complete_ Adam update rule (with the bias correction mechanism), not the first simplified version mentioned in the course notes. \n",
    "\n",
    "[2] Diederik Kingma and Jimmy Ba, \"Adam: A Method for Stochastic Optimization\", ICLR 2015."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 68
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 38885,
     "status": "ok",
     "timestamp": 1601234412016,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "ovUXV51Le1EE",
    "new_sheet": false,
    "outputId": "11c35e53-f9a3-4094-9b0a-485492bf3634",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Test Adam implementation\n",
    "from fully_connected_networks import adam\n",
    "\n",
    "reset_seed(0)\n",
    "\n",
    "N, D = 4, 5\n",
    "w = torch.linspace(-0.4, 0.6, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "dw = torch.linspace(-0.6, 0.4, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "m = torch.linspace(0.6, 0.9, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "v = torch.linspace(0.7, 0.5, steps=N*D, dtype=torch.float64, device='cuda').reshape(N, D)\n",
    "\n",
    "config = {'learning_rate': 1e-2, 'm': m, 'v': v, 't': 5}\n",
    "next_w, _ = adam(w, dw, config=config)\n",
    "\n",
    "expected_next_w = torch.tensor([\n",
    "  [-0.40094747, -0.34836187, -0.29577703, -0.24319299, -0.19060977],\n",
    "  [-0.1380274,  -0.08544591, -0.03286534,  0.01971428,  0.0722929],\n",
    "  [ 0.1248705,   0.17744702,  0.23002243,  0.28259667,  0.33516969],\n",
    "  [ 0.38774145,  0.44031188,  0.49288093,  0.54544852,  0.59801459]],\n",
    "   dtype=torch.float64, device='cuda')\n",
    "expected_v = torch.tensor([\n",
    "  [ 0.69966,     0.68908382,  0.67851319,  0.66794809,  0.65738853,],\n",
    "  [ 0.64683452,  0.63628604,  0.6257431,   0.61520571,  0.60467385,],\n",
    "  [ 0.59414753,  0.58362676,  0.57311152,  0.56260183,  0.55209767,],\n",
    "  [ 0.54159906,  0.53110598,  0.52061845,  0.51013645,  0.49966,   ]],\n",
    "   dtype=torch.float64, device='cuda')\n",
    "expected_m = torch.tensor([\n",
    "  [ 0.48,        0.49947368,  0.51894737,  0.53842105,  0.55789474],\n",
    "  [ 0.57736842,  0.59684211,  0.61631579,  0.63578947,  0.65526316],\n",
    "  [ 0.67473684,  0.69421053,  0.71368421,  0.73315789,  0.75263158],\n",
    "  [ 0.77210526,  0.79157895,  0.81105263,  0.83052632,  0.85      ]],\n",
    "   dtype=torch.float64, device='cuda')\n",
    "\n",
    "# You should see relative errors around e-7 or less\n",
    "print('next_w error: ', eecs598.grad.rel_error(expected_next_w, next_w))\n",
    "print('v error: ', eecs598.grad.rel_error(expected_v, config['v']))\n",
    "print('m error: ', eecs598.grad.rel_error(expected_m, config['m']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "1T_qzgxte1EI",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "Once you have debugged your RMSProp and Adam implementations, run the following to train a pair of deep networks using these new update rules:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 41551,
     "status": "ok",
     "timestamp": 1601234414686,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "6TFopQgre1EJ",
    "new_sheet": false,
    "outputId": "ee61b02f-531e-4b46-88f0-4a82936e5f9c",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "# Test Adam implementation\n",
    "from fully_connected_networks import adam, rmsprop, FullyConnectedNet\n",
    "\n",
    "for update_rule_name, update_rule_fn, learning_rate in [('adam', adam, 1e-3), ('rmsprop', rmsprop, 1e-4)]:\n",
    "  print('running with ', update_rule)\n",
    "  model = FullyConnectedNet([100, 100, 100, 100, 100], weight_scale=5e-2, device='cuda')\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=5, batch_size=100,\n",
    "                  update_rule=update_rule_fn,\n",
    "                  optim_config={\n",
    "                    'learning_rate': learning_rate\n",
    "                  },\n",
    "                  print_every=1000,\n",
    "                  verbose=True, device='cuda')\n",
    "  solvers[update_rule_name] = solver\n",
    "  solver.train()\n",
    "  print()\n",
    "\n",
    "plt.subplot(3, 1, 1)\n",
    "plt.title('Training loss')\n",
    "plt.xlabel('Iteration')\n",
    "for update_rule, solver in list(solvers.items()):\n",
    "  plt.plot(solver.loss_history, 'o', label=update_rule)\n",
    "plt.legend(loc='lower center', ncol=4)\n",
    "  \n",
    "plt.subplot(3, 1, 2)\n",
    "plt.title('Training accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "for update_rule, solver in list(solvers.items()):\n",
    "  plt.plot(solver.train_acc_history, '-o', label=update_rule)\n",
    "plt.legend(loc='lower center', ncol=4)\n",
    "\n",
    "plt.subplot(3, 1, 3)\n",
    "plt.title('Validation accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "for update_rule, solver in list(solvers.items()):\n",
    "  plt.plot(solver.val_acc_history, '-o', label=update_rule)\n",
    "plt.legend(loc='lower center', ncol=4)\n",
    "\n",
    "plt.gcf().set_size_inches(10, 20)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "C2_BL-2TwxKR",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    },
    "tags": [
     "pdf-title"
    ]
   },
   "source": [
    "# Dropout\n",
    "Dropout [1] is a technique for regularizing neural networks by randomly setting some output activations to zero during the forward pass. In this exercise you will implement a dropout layer and modify your fully-connected network to optionally use dropout.\n",
    "\n",
    "[1] [Geoffrey E. Hinton et al, \"Improving neural networks by preventing co-adaptation of feature detectors\", arXiv 2012](https://arxiv.org/abs/1207.0580)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "s68cb0QBwxKj",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Dropout: forward\n",
    "**Implement** the forward pass for dropout in `fully_connected_networks.py`. Since dropout behaves differently during training and testing, make sure to implement the operation for both modes.\n",
    "\n",
    "Run the following to test your dropout implementation. The mean of the output should be approximately the same during training and testing. During training the number of outputs set to zero should be approximately equal to the drop probability `p`, and during testing no outputs should be set to zero."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 374
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 41547,
     "status": "ok",
     "timestamp": 1601234414687,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "vFAmI9VxwxKk",
    "new_sheet": false,
    "outputId": "40b6df71-601e-4996-a329-e25947d63ec5",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import Dropout\n",
    "\n",
    "reset_seed(0)\n",
    "x = torch.randn(500, 500, dtype=torch.float64, device='cuda') + 10\n",
    "\n",
    "for p in [0.25, 0.4, 0.7]:\n",
    "  out, _ = Dropout.forward(x, {'mode': 'train', 'p': p})\n",
    "  out_test, _ = Dropout.forward(x, {'mode': 'test', 'p': p})\n",
    "\n",
    "  print('Running tests with p = ', p)\n",
    "  print('Mean of input: ', x.mean().item())\n",
    "  print('Mean of train-time output: ', out.mean().item())\n",
    "  print('Mean of test-time output: ', out_test.mean().item())\n",
    "  print('Fraction of train-time output set to zero: ', (out == 0).type(torch.float32).mean().item())\n",
    "  print('Fraction of test-time output set to zero: ', (out_test == 0).type(torch.float32).mean().item())\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "dt2BpwxswxKn",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Dropout: backward\n",
    "Implement the backward pass for dropout. After doing so, run the following cell to numerically gradient-check your implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 41543,
     "status": "ok",
     "timestamp": 1601234414687,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "3uctLwyIwxKo",
    "new_sheet": false,
    "outputId": "2085d569-b962-4d30-cbd4-45bcb06c3d17",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import Dropout\n",
    "\n",
    "reset_seed(0)\n",
    "x = torch.randn(10, 10, dtype=torch.float64, device='cuda') + 10\n",
    "dout = torch.randn_like(x)\n",
    "\n",
    "dropout_param = {'mode': 'train', 'p': 0.2, 'seed': 0}\n",
    "out, cache = Dropout.forward(x, dropout_param)\n",
    "dx = Dropout.backward(dout, cache)\n",
    "dx_num = eecs598.grad.compute_numeric_gradient(lambda xx: Dropout.forward(xx, dropout_param)[0], x, dout)\n",
    "\n",
    "# Error should be around e-10 or less\n",
    "print('dx relative error: ', eecs598.grad.rel_error(dx, dx_num))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "OLzMLx-iwxKs",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "# Fully-connected nets with dropout\n",
    "Modify [your implementation](#scrollTo=7p-goSyucyZH) of `FullyConnectedNet` to use dropout. Specifically, if the constructor of the network receives a value that is not 0 for the `dropout` parameter, then the net should add a dropout layer immediately after every ReLU nonlinearity.\n",
    "\n",
    "After doing so, run the following to numerically gradient-check your implementation. You should see errors less than `1e-5`, and different dropout rates should result different error values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 476
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 51857,
     "status": "ok",
     "timestamp": 1601234425005,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "18ugsX0iwxKu",
    "new_sheet": false,
    "outputId": "ea4c0a94-61a1-483b-9d73-e0ccc7c350e0",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import FullyConnectedNet\n",
    "\n",
    "reset_seed(0)\n",
    "\n",
    "N, D, H1, H2, C = 2, 15, 20, 30, 10\n",
    "X = torch.randn(N, D, dtype=torch.float64, device='cuda')\n",
    "y = torch.randint(C, size=(N,), dtype=torch.int64, device='cuda')\n",
    "\n",
    "for dropout in [0, 0.25, 0.5]:\n",
    "  print('Running check with dropout = ', dropout)\n",
    "  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,\n",
    "                            weight_scale=5e-2, dropout=dropout,\n",
    "                            seed=0, dtype=torch.float64, device='cuda')\n",
    "\n",
    "  loss, grads = model.loss(X, y)\n",
    "  print('Initial loss: ', loss.item())\n",
    "  \n",
    "  # Relative errors should be around e-5 or less.\n",
    "  for name in sorted(grads):\n",
    "    f = lambda _: model.loss(X, y)[0]\n",
    "    grad_num = eecs598.grad.compute_numeric_gradient(f, model.params[name])\n",
    "    print('%s relative error: %.2e' % (name, eecs598.grad.rel_error(grad_num, grads[name])))\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "dmhrgg5hwxKy",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "## Regularization experiment\n",
    "To get a sense of the way that dropout can regularize a neural network, we will train three different two-layer networks:\n",
    "\n",
    "1. Hidden size 256, dropout = 0\n",
    "2. Hidden size 512, dropout = 0\n",
    "3. Hidden size 512, dropout = 0.5\n",
    "\n",
    "We will then visualize the training and validation accuracies of these three networks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 731
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 91056,
     "status": "ok",
     "timestamp": 1601234464209,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "N6srh4BLwxKz",
    "new_sheet": false,
    "outputId": "0ec90ee1-c74c-4178-97ee-f477fb3bd678",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "from fully_connected_networks import FullyConnectedNet\n",
    "\n",
    "# Train two identical nets, one with dropout and one without\n",
    "reset_seed(0)\n",
    "num_train = 20000\n",
    "small_data = {\n",
    "  'X_train': data_dict['X_train'][:num_train],\n",
    "  'y_train': data_dict['y_train'][:num_train],\n",
    "  'X_val': data_dict['X_val'],\n",
    "  'y_val': data_dict['y_val'],\n",
    "}\n",
    "\n",
    "solvers = {}\n",
    "dropout_choices = [0, 0, 0.5]\n",
    "width_choices = [256, 512, 512]\n",
    "for dropout, width in zip(dropout_choices, width_choices):\n",
    "# for dropout in dropout_choices:\n",
    "  model = FullyConnectedNet([width], dropout=dropout, dtype=torch.float32, device='cuda')\n",
    "  print('Training a model with dropout=%.2f and width=%d' % (dropout, width))\n",
    "\n",
    "  solver = Solver(model, small_data,\n",
    "                  num_epochs=100, batch_size=512,\n",
    "                  update_rule=adam,\n",
    "                  optim_config={\n",
    "                    'learning_rate': 5e-3,\n",
    "                  },\n",
    "                  print_every=100000, print_acc_every=10,\n",
    "                  verbose=True, device='cuda')\n",
    "  solver.train()\n",
    "  solvers[(dropout, width)] = solver\n",
    "  print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "button": false,
    "deletable": true,
    "id": "mjdKh3apRtX4",
    "new_sheet": false,
    "run_control": {
     "read_only": false
    }
   },
   "source": [
    "If everything worked as expected, you should see that the network with dropout has lower training accuracies than the networks without dropout, but that it achieves higher validation accuracies.\n",
    "\n",
    "You should also see that a network with width 512 and dropout 0.5 achieves higher validation accuracies than a network with width 256 and no dropout. This demonstrates that reducing the model size is not generally an effective regularization strategy -- it's often better to use a larger model with explicit regularization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "button": false,
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 908
    },
    "deletable": true,
    "executionInfo": {
     "elapsed": 91541,
     "status": "ok",
     "timestamp": 1601234464698,
     "user": {
      "displayName": "Mohamed El Banani",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14GggOGiYfSuGtZ3nDZTHgI1FZ7khEDL9VZbcpo8=s64",
      "userId": "10640812476884023238"
     },
     "user_tz": 240
    },
    "id": "aCDhFCR0wxK2",
    "new_sheet": false,
    "outputId": "d52fefe3-559f-4a10-d3fc-815c3f489c35",
    "run_control": {
     "read_only": false
    }
   },
   "outputs": [],
   "source": [
    "plt.subplot(2, 1, 1)\n",
    "for (dropout, width), solver in solvers.items():\n",
    "  train_acc = solver.train_acc_history\n",
    "  label = 'dropout=%.2f, width=%d' % (dropout, width)\n",
    "  plt.plot(train_acc, 'o', label=label)\n",
    "plt.title('Train accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "  \n",
    "plt.subplot(2, 1, 2)\n",
    "for (dropout, width), solver in solvers.items():\n",
    "  val_acc = solver.val_acc_history\n",
    "  label = 'dropout=%.2f, width=%d' % (dropout, width)\n",
    "  plt.plot(val_acc, 'o', label=label)\n",
    "plt.ylim(0.4, 0.52)\n",
    "plt.title('Val accuracy')\n",
    "plt.xlabel('Epoch')\n",
    "plt.ylabel('Accuracy')\n",
    "plt.legend(ncol=2, loc='lower right')\n",
    "\n",
    "plt.gcf().set_size_inches(10, 15)\n",
    "plt.show()"
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "fully_connected_networks.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}