loss_functions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Loss Functions\n",
    "Given an input vector, a loss function is a measure of how bad a particular model performs in predicting a desired output  quantity (regression) or correctly labeling the input vector (classification).\n",
    "\n",
    "\n",
    "Farhad Kamangar  Sept. 2018"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mean Squared Error (MSE).\n",
    "\n",
    "Mean Squared Error is the most commonly used regression loss function and it is defined as the mean of the squared distances between the desired and the predicted output(s).\n",
    "\n",
    "$$\\large MSE=\\frac{1}{N}\\sum_{i=1}^{N} {(t_i-y_i)}^2$$\n",
    "\n",
    "where $N$ is the total number of samples , $t_i$ is the desired output for sample $i$ and $y_i$ is the actual output for sample $i$\n",
    "\n",
    "\n",
    "## Mean Absolute Error (MAE).\n",
    "\n",
    "Mean Absolute Error is another common function used for regression models and it is defined as as the mean of the absolute differences between the desired and the predicted output(s).\n",
    "\n",
    "$$\\large MAE=\\frac{1}{N}\\sum_{i=1}^{N} {|t_i-y_i|}$$\n",
    "\n",
    "where $N$ is the total number of samples , $t_i$ is the desired output for sample $i$ and $y_i$ is the actual output for sample $i$\n",
    "\n",
    "\n",
    "\n",
    "## Hinge (Multiclass Support Vector Machine) Loss\n",
    "\n",
    "The hinge loss function is used for classification and it is based on the concept of maximum-margin. The hinge loss is formulated as:\n",
    "\n",
    "$$\\large L=\\frac {1}{N} \\sum_{i} L_i$$\n",
    "\n",
    "$$\\large L_i=\\sum_{j \\neq i} max(0,y_j-y_i+\\Delta)$$\n",
    "\n",
    "where $N$ is the number of the samples,  $i$ is the index of the true class, and $\\Delta$ is a constant.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Numerical Example"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actual output:  [  9.   7.  11.]\n",
      "Loss:\n",
      "  0.0\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "def SVM_loss(input_vector, w,true_class_index,delta=1):\n",
    "    \"\"\"\n",
    "    This function calculated the hinge loss function\n",
    "    Farhad Kamangar Sept. 2018\n",
    "    \"\"\"\n",
    "    y = np.dot(w,input_vector)\n",
    "    print(\"Actual output: \",y)\n",
    "    margins = np.maximum(0, y - y[true_class_index] + delta)\n",
    "    margins[true_class_index] = 0\n",
    "    loss_i = np.sum(margins)\n",
    "    return loss_i\n",
    "\n",
    "x =np.transpose( np.array([[1.0, 1.0, 1], [1.0, 0,0], [0,1,0]]))\n",
    "true_class_index=[0,2,1]\n",
    "w=np.array([[2,4,7], [1,5,6], [6,2,5]])\n",
    "index=1\n",
    "loss=SVM_loss(x[index], w,true_class_index[index],delta=1)\n",
    "print(\"Loss:\\n \", loss)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cross Entropy Loss\n",
    "\n",
    "The cross entropy loss uses a softmax function to calculate the loss. \n",
    "\n",
    "### Softmax Function\n",
    "\n",
    "The softmax function gives a probabilistic interpretation to the output values and it is formulated as:\n",
    "\n",
    "$$\\large S(y_i)=\\frac{e^{y_i}}{\\sum_{j}e^{y_j}}$$\n",
    "\n",
    "This function interprets the outputs as unnormalized log probabilities of each class. Notice that the denominator of the above equation normalizes the probabilities so the total sums to 1. \n",
    "\n",
    "In other words the softmax function takes a vector of floating point numbers and proportionally compresses each number between zero and one such that the total adds up to 1.\n",
    "\n",
    "Using the softmax function, the cross entropy loss can be calculated as:\n",
    "\n",
    "$$\\large L_i=-log(\\frac{e^{y_i}}{\\sum_{j}e^{y_j}})$$\n",
    "\n",
    "\n",
    "The above equation is really a simplified version of a discrete cross entropy between two distributions.\n",
    "\n",
    "Let's imagine that we have a true discrete distribution $p$ and an estimated discrete distribution $q$. The cross entropy between these two distribution is defined as:\n",
    "\n",
    "$$\\large H(p,q)= - \\sum_{x}p(x)log(q(x))$$\n",
    "\n",
    "Notice that in a multi-class classification problem the true probability distribution has all zeros except for the correct class, $i$, which has the value of 1:\n",
    "\n",
    "$$\\large p=[0,0,..., 1, ... 0]$$\n",
    "\n",
    "If the above discrete distribution $p$ is substituted into the general cross entropy equation it will result in the simplified cross entropy loss \n",
    "\n",
    "$$\\large L_i=-log(\\frac{e^{y_i}}{\\sum_{j}e^{y_j}})$$\n",
    "\n",
    "where $i$ is the index of the correct class.\n",
    "\n",
    "Notice that to calculate the overall loss we still need to average the loss over all the samples.\n",
    "\n",
    "$$\\large L=\\frac {1}{Q}\\sum_{k=1}^{Q}{L_k}$$\n",
    "\n",
    "where Q is the number of samples and $l_k$ is the cross entropy loss for sample $k$\n",
    "\n",
    "**Note:** There is no \"softmax loss\". The correct terminology is \"cross-entropy loss\". The \"cross entropy loss\" uses the \"softmax\" function to calculate the loss.  "
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}