{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional Lab: Feature scaling and Learning Rate (Multi-variable)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goals\n",
    "In this lab you will:\n",
    "- Utilize  the multiple variables routines developed in the previous lab\n",
    "- run Gradient Descent on a data set with multiple features\n",
    "- explore the impact of the *learning rate alpha* on gradient descent\n",
    "- improve performance of gradient descent by *feature scaling* using z-score normalization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tools\n",
    "You will utilize the functions developed in the last lab as well as matplotlib and NumPy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from lab_utils_multi import  load_house_data, run_gradient_descent \n",
    "from lab_utils_multi import  norm_plot, plt_equal_scale, plot_cost_i_w\n",
    "from lab_utils_common import dlc\n",
    "np.set_printoptions(precision=2)\n",
    "plt.style.use('./deeplearning.mplstyle')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notation\n",
    "\n",
    "|General <br />  Notation  | Description| Python (if applicable) |\n",
    "|: ------------|: ------------------------------------------------------------||\n",
    "| $a$ | scalar, non bold                                                      ||\n",
    "| $\\mathbf{a}$ | vector, bold                                                 ||\n",
    "| $\\mathbf{A}$ | matrix, bold capital                                         ||\n",
    "| **Regression** |         |    |     |\n",
    "|  $\\mathbf{X}$ | training example maxtrix                  | `X_train` |   \n",
    "|  $\\mathbf{y}$  | training example  targets                | `y_train` \n",
    "|  $\\mathbf{x}^{(i)}$, $y^{(i)}$ | $i_{th}$Training Example | `X[i]`, `y[i]`|\n",
    "| m | number of training examples | `m`|\n",
    "| n | number of features in each example | `n`|\n",
    "|  $\\mathbf{w}$  |  parameter: weight,                       | `w`    |\n",
    "|  $b$           |  parameter: bias                                           | `b`    |     \n",
    "| $f_{\\mathbf{w},b}(\\mathbf{x}^{(i)})$ | The result of the model evaluation at  $\\mathbf{x}^{(i)}$ parameterized by $\\mathbf{w},b$: $f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) = \\mathbf{w} \\cdot \\mathbf{x}^{(i)}+b$  | `f_wb` | \n",
    "|$\\frac{\\partial J(\\mathbf{w},b)}{\\partial w_j}$| the gradient or partial derivative of cost with respect to a parameter $w_j$ |`dj_dw[j]`| \n",
    "|$\\frac{\\partial J(\\mathbf{w},b)}{\\partial b}$| the gradient or partial derivative of cost with respect to a parameter $b$| `dj_db`|"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Problem Statement\n",
    "\n",
    "As in the previous labs, you will use the motivating example of housing price prediction. The training data set contains many examples with 4 features (size, bedrooms, floors and age) shown in the table below. Note, in this lab, the Size feature is in sqft while earlier labs utilized 1000 sqft.  This data set is larger than the previous lab.\n",
    "\n",
    "We would like to build a linear regression model using these values so we can then predict the price for other houses - say, a house with 1200 sqft, 3 bedrooms, 1 floor, 40 years old. \n",
    "\n",
    "##  Dataset: \n",
    "| Size (sqft) | Number of Bedrooms  | Number of floors | Age of  Home | Price (1000s dollars)  |   \n",
    "| ----------------| ------------------- |----------------- |--------------|----------------------- |  \n",
    "| 952             | 2                   | 1                | 65           | 271.5                  |  \n",
    "| 1244            | 3                   | 2                | 64           | 232                    |  \n",
    "| 1947            | 3                   | 2                | 17           | 509.8                  |  \n",
    "| ...             | ...                 | ...              | ...          | ...                    |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset\n",
    "X_train, y_train = load_house_data()\n",
    "X_features = ['size(sqft)','bedrooms','floors','age']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's view the dataset and its features by plotting each feature versus price."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 26772 characters]\n",
      "text/plain": [
       "<Figure size 864x216 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig,ax=plt.subplots(1, 4, figsize=(12, 3), sharey=True)\n",
    "for i in range(len(ax)):\n",
    "    ax[i].scatter(X_train[:,i],y_train)\n",
    "    ax[i].set_xlabel(X_features[i])\n",
    "ax[0].set_ylabel(\"Price (1000's)\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Plotting each feature vs. the target, price, provides some indication of which features have the strongest influence on price. Above, increasing size also increases price. Bedrooms and floors don't seem to have a strong impact on price. Newer houses have higher prices than older houses."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name=\"toc_15456_5\"></a>\n",
    "## Gradient Descent With Multiple Variables\n",
    "Here are the equations you developed in the last lab on gradient descent for multiple variables.:\n",
    "\n",
    "$$\\begin{align*} \\text{repeat}&\\text{ until convergence:} \\; \\lbrace \\newline\\;\n",
    "& w_j := w_j -  \\alpha \\frac{\\partial J(\\mathbf{w},b)}{\\partial w_j} \\tag{1}  \\; & \\text{for j = 0..n-1}\\newline\n",
    "&b\\ \\ := b -  \\alpha \\frac{\\partial J(\\mathbf{w},b)}{\\partial b}  \\newline \\rbrace\n",
    "\\end{align*}$$\n",
    "\n",
    "where, n is the number of features, parameters $w_j$,  $b$, are updated simultaneously and where  \n",
    "\n",
    "$$\n",
    "\\begin{align}\n",
    "\\frac{\\partial J(\\mathbf{w},b)}{\\partial w_j}  &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)})x_{j}^{(i)} \\tag{2}  \\\\\n",
    "\\frac{\\partial J(\\mathbf{w},b)}{\\partial b}  &= \\frac{1}{m} \\sum\\limits_{i = 0}^{m-1} (f_{\\mathbf{w},b}(\\mathbf{x}^{(i)}) - y^{(i)}) \\tag{3}\n",
    "\\end{align}\n",
    "$$\n",
    "* m is the number of training examples in the data set\n",
    "\n",
    "    \n",
    "*  $f_{\\mathbf{w},b}(\\mathbf{x}^{(i)})$ is the model's prediction, while $y^{(i)}$ is the target value\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Rate\n",
    "<figure>\n",
    "    <img src=\"./images/C1_W2_Lab06_learningrate.PNG\" style=\"width:1200px;\" >\n",
    "</figure>\n",
    "The lectures discussed some of the issues related to setting the learning rate $\\alpha$. The learning rate controls the size of the update to the parameters. See equation (1) above. It is shared by all the parameters.  \n",
    "\n",
    "Let's run gradient descent and try a few settings of $\\alpha$ on our data set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\alpha$ = 9.9e-7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration Cost          w0       w1       w2       w3       b       djdw0    djdw1    djdw2    djdw3    djdb  \n",
      "---------------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n",
      "        0 9.55884e+04  5.5e-01  1.0e-03  5.1e-04  1.2e-02  3.6e-04 -5.5e+05 -1.0e+03 -5.2e+02 -1.2e+04 -3.6e+02\n",
      "        1 1.28213e+05 -8.8e-02 -1.7e-04 -1.0e-04 -3.4e-03 -4.8e-05  6.4e+05  1.2e+03  6.2e+02  1.6e+04  4.1e+02\n",
      "        2 1.72159e+05  6.5e-01  1.2e-03  5.9e-04  1.3e-02  4.3e-04 -7.4e+05 -1.4e+03 -7.0e+02 -1.7e+04 -4.9e+02\n",
      "        3 2.31358e+05 -2.1e-01 -4.0e-04 -2.3e-04 -7.5e-03 -1.2e-04  8.6e+05  1.6e+03  8.3e+02  2.1e+04  5.6e+02\n",
      "        4 3.11100e+05  7.9e-01  1.4e-03  7.1e-04  1.5e-02  5.3e-04 -1.0e+06 -1.8e+03 -9.5e+02 -2.3e+04 -6.6e+02\n",
      "        5 4.18517e+05 -3.7e-01 -7.1e-04 -4.0e-04 -1.3e-02 -2.1e-04  1.2e+06  2.1e+03  1.1e+03  2.8e+04  7.5e+02\n",
      "        6 5.63212e+05  9.7e-01  1.7e-03  8.7e-04  1.8e-02  6.6e-04 -1.3e+06 -2.5e+03 -1.3e+03 -3.1e+04 -8.8e+02\n",
      "        7 7.58122e+05 -5.8e-01 -1.1e-03 -6.2e-04 -1.9e-02 -3.4e-04  1.6e+06  2.9e+03  1.5e+03  3.8e+04  1.0e+03\n",
      "        8 1.02068e+06  1.2e+00  2.2e-03  1.1e-03  2.3e-02  8.3e-04 -1.8e+06 -3.3e+03 -1.7e+03 -4.2e+04 -1.2e+03\n",
      "        9 1.37435e+06 -8.7e-01 -1.7e-03 -9.1e-04 -2.7e-02 -5.2e-04  2.1e+06  3.9e+03  2.0e+03  5.1e+04  1.4e+03\n",
      "w,b found by gradient descent: w: [-0.87 -0.   -0.   -0.03], b: -0.00\n"
     ]
    }
   ],
   "source": [
    "#set alpha to 9.9e-7\n",
    "_, _, hist = run_gradient_descent(X_train, y_train, 10, alpha = 9.9e-7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears the learning rate is too high.  The solution does not converge. Cost is *increasing* rather than decreasing. Let's plot the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 35272 characters]\n",
      "text/plain": [
       "<Figure size 864x216 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_cost_i_w(X_train, y_train, hist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot on the right shows the value of one of the parameters, $w_0$. At each iteration, it is overshooting the optimal value and as a result, cost ends up *increasing* rather than approaching the minimum. Note that this is not a completely accurate picture as there are 4 parameters being modified each pass rather than just one. This plot is only showing $w_0$ with the other parameters fixed at benign values. In this and later plots you may notice the blue and orange lines being slightly off."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### $\\alpha$ = 9e-7\n",
    "Let's try a bit smaller value and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration Cost          w0       w1       w2       w3       b       djdw0    djdw1    djdw2    djdw3    djdb  \n",
      "---------------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n",
      "        0 6.64616e+04  5.0e-01  9.1e-04  4.7e-04  1.1e-02  3.3e-04 -5.5e+05 -1.0e+03 -5.2e+02 -1.2e+04 -3.6e+02\n",
      "        1 6.18990e+04  1.8e-02  2.1e-05  2.0e-06 -7.9e-04  1.9e-05  5.3e+05  9.8e+02  5.2e+02  1.3e+04  3.4e+02\n",
      "        2 5.76572e+04  4.8e-01  8.6e-04  4.4e-04  9.5e-03  3.2e-04 -5.1e+05 -9.3e+02 -4.8e+02 -1.1e+04 -3.4e+02\n",
      "        3 5.37137e+04  3.4e-02  3.9e-05  2.8e-06 -1.6e-03  3.8e-05  4.9e+05  9.1e+02  4.8e+02  1.2e+04  3.2e+02\n",
      "        4 5.00474e+04  4.6e-01  8.2e-04  4.1e-04  8.0e-03  3.2e-04 -4.8e+05 -8.7e+02 -4.5e+02 -1.1e+04 -3.1e+02\n",
      "        5 4.66388e+04  5.0e-02  5.6e-05  2.5e-06 -2.4e-03  5.6e-05  4.6e+05  8.5e+02  4.5e+02  1.2e+04  2.9e+02\n",
      "        6 4.34700e+04  4.5e-01  7.8e-04  3.8e-04  6.4e-03  3.2e-04 -4.4e+05 -8.1e+02 -4.2e+02 -9.8e+03 -2.9e+02\n",
      "        7 4.05239e+04  6.4e-02  7.0e-05  1.2e-06 -3.3e-03  7.3e-05  4.3e+05  7.9e+02  4.2e+02  1.1e+04  2.7e+02\n",
      "        8 3.77849e+04  4.4e-01  7.5e-04  3.5e-04  4.9e-03  3.2e-04 -4.1e+05 -7.5e+02 -3.9e+02 -9.1e+03 -2.7e+02\n",
      "        9 3.52385e+04  7.7e-02  8.3e-05 -1.1e-06 -4.2e-03  8.9e-05  4.0e+05  7.4e+02  3.9e+02  1.0e+04  2.5e+02\n",
      "w,b found by gradient descent: w: [ 7.74e-02  8.27e-05 -1.06e-06 -4.20e-03], b: 0.00\n"
     ]
    }
   ],
   "source": [
    "#set alpha to 9e-7\n",
    "_,_,hist = run_gradient_descent(X_train, y_train, 10, alpha = 9e-7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cost is decreasing throughout the run showing that alpha is not too large. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 34916 characters]\n",
      "text/plain": [
       "<Figure size 864x216 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_cost_i_w(X_train, y_train, hist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the left, you see that cost is decreasing as it should. On the right, you can see that $w_0$ is still oscillating around the minimum, but it is decreasing each iteration rather than increasing. Note above that `dj_dw[0]` changes sign with each iteration as `w[0]` jumps over the optimal value.\n",
    "This alpha value will converge. You can vary the number of iterations to see how it behaves."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### $\\alpha$ = 1e-7\n",
    "Let's try a bit smaller value for $\\alpha$ and see what happens."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration Cost          w0       w1       w2       w3       b       djdw0    djdw1    djdw2    djdw3    djdb  \n",
      "---------------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n",
      "        0 4.42313e+04  5.5e-02  1.0e-04  5.2e-05  1.2e-03  3.6e-05 -5.5e+05 -1.0e+03 -5.2e+02 -1.2e+04 -3.6e+02\n",
      "        1 2.76461e+04  9.8e-02  1.8e-04  9.2e-05  2.2e-03  6.5e-05 -4.3e+05 -7.9e+02 -4.0e+02 -9.5e+03 -2.8e+02\n",
      "        2 1.75102e+04  1.3e-01  2.4e-04  1.2e-04  2.9e-03  8.7e-05 -3.4e+05 -6.1e+02 -3.1e+02 -7.3e+03 -2.2e+02\n",
      "        3 1.13157e+04  1.6e-01  2.9e-04  1.5e-04  3.5e-03  1.0e-04 -2.6e+05 -4.8e+02 -2.4e+02 -5.6e+03 -1.8e+02\n",
      "        4 7.53002e+03  1.8e-01  3.3e-04  1.7e-04  3.9e-03  1.2e-04 -2.1e+05 -3.7e+02 -1.9e+02 -4.2e+03 -1.4e+02\n",
      "        5 5.21639e+03  2.0e-01  3.5e-04  1.8e-04  4.2e-03  1.3e-04 -1.6e+05 -2.9e+02 -1.5e+02 -3.1e+03 -1.1e+02\n",
      "        6 3.80242e+03  2.1e-01  3.8e-04  1.9e-04  4.5e-03  1.4e-04 -1.3e+05 -2.2e+02 -1.1e+02 -2.3e+03 -8.6e+01\n",
      "        7 2.93826e+03  2.2e-01  3.9e-04  2.0e-04  4.6e-03  1.4e-04 -9.8e+04 -1.7e+02 -8.6e+01 -1.7e+03 -6.8e+01\n",
      "        8 2.41013e+03  2.3e-01  4.1e-04  2.1e-04  4.7e-03  1.5e-04 -7.7e+04 -1.3e+02 -6.5e+01 -1.2e+03 -5.4e+01\n",
      "        9 2.08734e+03  2.3e-01  4.2e-04  2.1e-04  4.8e-03  1.5e-04 -6.0e+04 -1.0e+02 -4.9e+01 -7.5e+02 -4.3e+01\n",
      "w,b found by gradient descent: w: [2.31e-01 4.18e-04 2.12e-04 4.81e-03], b: 0.00\n"
     ]
    }
   ],
   "source": [
    "#set alpha to 1e-7\n",
    "_,_,hist = run_gradient_descent(X_train, y_train, 10, alpha = 1e-7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Cost is decreasing throughout the run showing that $\\alpha$ is not too large. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 33576 characters]\n",
      "text/plain": [
       "<Figure size 864x216 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_cost_i_w(X_train,y_train,hist)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On the left, you see that cost is decreasing as it should. On the right you can see that $w_0$ is decreasing without crossing the minimum. Note above that `dj_w0` is negative throughout the run. This solution will also converge, though not quite as quickly as the previous example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Feature Scaling \n",
    "<figure>\n",
    "    <img src=\"./images/C1_W2_Lab06_featurescalingheader.PNG\" style=\"width:1200px;\" >\n",
    "</figure>\n",
    "The lectures described the importance of rescaling the dataset so the features have a similar range.\n",
    "If you are interested in the details of why this is the case, click on the 'details' header below. If not, the section below will walk through an implementation of how to do feature scaling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<details>\n",
    "<summary>\n",
    "    <font size='3', color='darkgreen'><b>Details</b></font>\n",
    "</summary>\n",
    "\n",
    "Let's look again at the situation with $\\alpha$ = 9e-7. This is pretty close to the maximum value we can set $\\alpha$  to without diverging. This is a short run showing the first few iterations:\n",
    "\n",
    "<figure>\n",
    "    <img src=\"./images/C1_W2_Lab06_ShortRun.PNG\" style=\"width:1200px;\" >\n",
    "</figure>\n",
    "\n",
    "Above, while cost is being decreased, its clear that $w_0$ is making more rapid progress than the other parameters due to its much larger gradient.\n",
    "\n",
    "The graphic below shows the result of a very long run with $\\alpha$ = 9e-7. This takes several hours.\n",
    "\n",
    "<figure>\n",
    "    <img src=\"./images/C1_W2_Lab06_LongRun.PNG\" style=\"width:1200px;\" >\n",
    "</figure>\n",
    "    \n",
    "Above, you can see cost decreased slowly after its initial reduction. Notice the difference between `w0` and `w1`,`w2`,`w3` as well as  `dj_dw0` and `dj_dw1-3`. `w0` reaches its near final value very quickly and `dj_dw0` has quickly decreased to a small value showing that `w0` is near the final value. The other parameters were reduced much more slowly.\n",
    "\n",
    "Why is this?  Is there something we can improve? See below:\n",
    "<figure>\n",
    "    <center> <img src=\"./images/C1_W2_Lab06_scale.PNG\"   ></center>\n",
    "</figure>   \n",
    "\n",
    "The figure above shows why $w$'s are updated unevenly. \n",
    "- $\\alpha$ is shared by all parameter updates ($w$'s and $b$).\n",
    "- the common error term is multiplied by the features for the $w$'s. (not $b$).\n",
    "- the features vary significantly in magnitude making some features update much faster than others. In this case, $w_0$ is multiplied by 'size(sqft)', which is generally > 1000,  while $w_1$ is multiplied by 'number of bedrooms', which is generally 2-4. \n",
    "    \n",
    "The solution is Feature Scaling."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The lectures discussed three different techniques: \n",
    "- Feature scaling, essentially dividing each positive feature by its maximum value, or more generally, rescale each feature by both its minimum and maximum values using (x-min)/(max-min). Both ways normalizes features to the range of -1 and 1, where the former method works for positive features which is simple and serves well for the lecture's example, and the latter method works for any features.\n",
    "- Mean normalization: $x_i := \\dfrac{x_i - \\mu_i}{max - min} $ \n",
    "- Z-score normalization which we will explore below. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### z-score normalization \n",
    "After z-score normalization, all features will have a mean of 0 and a standard deviation of 1.\n",
    "\n",
    "To implement z-score normalization, adjust your input values as shown in this formula:\n",
    "$$x^{(i)}_j = \\dfrac{x^{(i)}_j - \\mu_j}{\\sigma_j} \\tag{4}$$ \n",
    "where $j$ selects a feature or a column in the $\\mathbf{X}$ matrix. $µ_j$ is the mean of all the values for feature (j) and $\\sigma_j$ is the standard deviation of feature (j).\n",
    "$$\n",
    "\\begin{align}\n",
    "\\mu_j &= \\frac{1}{m} \\sum_{i=0}^{m-1} x^{(i)}_j \\tag{5}\\\\\n",
    "\\sigma^2_j &= \\frac{1}{m} \\sum_{i=0}^{m-1} (x^{(i)}_j - \\mu_j)^2  \\tag{6}\n",
    "\\end{align}\n",
    "$$\n",
    "\n",
    ">**Implementation Note:** When normalizing the features, it is important\n",
    "to store the values used for normalization - the mean value and the standard deviation used for the computations. After learning the parameters\n",
    "from the model, we often want to predict the prices of houses we have not\n",
    "seen before. Given a new x value (living room area and number of bed-\n",
    "rooms), we must first normalize x using the mean and standard deviation\n",
    "that we had previously computed from the training set.\n",
    "\n",
    "**Implementation**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def zscore_normalize_features(X):\n",
    "    \"\"\"\n",
    "    computes  X, zcore normalized by column\n",
    "    \n",
    "    Args:\n",
    "      X (ndarray (m,n))     : input data, m examples, n features\n",
    "      \n",
    "    Returns:\n",
    "      X_norm (ndarray (m,n)): input normalized by column\n",
    "      mu (ndarray (n,))     : mean of each feature\n",
    "      sigma (ndarray (n,))  : standard deviation of each feature\n",
    "    \"\"\"\n",
    "    # find the mean of each column/feature\n",
    "    mu     = np.mean(X, axis=0)                 # mu will have shape (n,)\n",
    "    # find the standard deviation of each column/feature\n",
    "    sigma  = np.std(X, axis=0)                  # sigma will have shape (n,)\n",
    "    # element-wise, subtract mu for that column from each example, divide by std for that column\n",
    "    X_norm = (X - mu) / sigma      \n",
    "\n",
    "    return (X_norm, mu, sigma)\n",
    " \n",
    "#check our work\n",
    "#from sklearn.preprocessing import scale\n",
    "#scale(X_orig, axis=0, with_mean=True, with_std=True, copy=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the steps involved in Z-score normalization. The plot below shows the transformation step by step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28736 characters]\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "mu     = np.mean(X_train,axis=0)   \n",
    "sigma  = np.std(X_train,axis=0) \n",
    "X_mean = (X_train - mu)\n",
    "X_norm = (X_train - mu)/sigma      \n",
    "\n",
    "fig,ax=plt.subplots(1, 3, figsize=(12, 3))\n",
    "ax[0].scatter(X_train[:,0], X_train[:,3])\n",
    "ax[0].set_xlabel(X_features[0]); ax[0].set_ylabel(X_features[3]);\n",
    "ax[0].set_title(\"unnormalized\")\n",
    "ax[0].axis('equal')\n",
    "\n",
    "ax[1].scatter(X_mean[:,0], X_mean[:,3])\n",
    "ax[1].set_xlabel(X_features[0]); ax[0].set_ylabel(X_features[3]);\n",
    "ax[1].set_title(r\"X - $\\mu$\")\n",
    "ax[1].axis('equal')\n",
    "\n",
    "ax[2].scatter(X_norm[:,0], X_norm[:,3])\n",
    "ax[2].set_xlabel(X_features[0]); ax[0].set_ylabel(X_features[3]);\n",
    "ax[2].set_title(r\"Z-score normalized\")\n",
    "ax[2].axis('equal')\n",
    "plt.tight_layout(rect=[0, 0.03, 1, 0.95])\n",
    "fig.suptitle(\"distribution of features before, during, after normalization\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The plot above shows the relationship between two of the training set parameters, \"age\" and \"size(sqft)\". *These are plotted with equal scale*. \n",
    "- Left: Unnormalized: The range of values or the variance of the 'size(sqft)' feature is much larger than that of age\n",
    "- Middle: The first step removes the mean or average value from each feature. This leaves features that are centered around zero. It's difficult to see the difference for the 'age' feature, but 'size(sqft)' is clearly around zero.\n",
    "- Right: The second step divides by the standard deviation. This leaves both features centered at zero with a similar scale."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's normalize the data and compare it to the original data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_mu = [1.42e+03 2.72e+00 1.38e+00 3.84e+01], \n",
      "X_sigma = [411.62   0.65   0.49  25.78]\n",
      "Peak to Peak range by column in Raw        X:[2.41e+03 4.00e+00 1.00e+00 9.50e+01]\n",
      "Peak to Peak range by column in Normalized X:[5.85 6.14 2.06 3.69]\n"
     ]
    }
   ],
   "source": [
    "# normalize the original features\n",
    "X_norm, X_mu, X_sigma = zscore_normalize_features(X_train)\n",
    "print(f\"X_mu = {X_mu}, \\nX_sigma = {X_sigma}\")\n",
    "print(f\"Peak to Peak range by column in Raw        X:{np.ptp(X_train,axis=0)}\")   \n",
    "print(f\"Peak to Peak range by column in Normalized X:{np.ptp(X_norm,axis=0)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The peak to peak range of each column is reduced from a factor of thousands to a factor of 2-3 by normalization."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 39156 characters]\n",
      "text/plain": [
       "<Figure size 864x216 with 8 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "[encoded data, 38484 characters]\n",
      "text/plain": [
       "<Figure size 864x216 with 8 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig,ax=plt.subplots(1, 4, figsize=(12, 3))\n",
    "for i in range(len(ax)):\n",
    "    norm_plot(ax[i],X_train[:,i],)\n",
    "    ax[i].set_xlabel(X_features[i])\n",
    "ax[0].set_ylabel(\"count\");\n",
    "fig.suptitle(\"distribution of features before normalization\")\n",
    "plt.show()\n",
    "fig,ax=plt.subplots(1,4,figsize=(12,3))\n",
    "for i in range(len(ax)):\n",
    "    norm_plot(ax[i],X_norm[:,i],)\n",
    "    ax[i].set_xlabel(X_features[i])\n",
    "ax[0].set_ylabel(\"count\"); \n",
    "fig.suptitle(\"distribution of features after normalization\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice, above, the range of the normalized data (x-axis) is centered around zero and roughly +/- 2. Most importantly, the range is similar for each feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's re-run our gradient descent algorithm with normalized data.\n",
    "Note the **vastly larger value of alpha**. This will speed up gradient descent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration Cost          w0       w1       w2       w3       b       djdw0    djdw1    djdw2    djdw3    djdb  \n",
      "---------------------|--------|--------|--------|--------|--------|--------|--------|--------|--------|--------|\n",
      "        0 5.76170e+04  8.9e+00  3.0e+00  3.3e+00 -6.0e+00  3.6e+01 -8.9e+01 -3.0e+01 -3.3e+01  6.0e+01 -3.6e+02\n",
      "      100 2.21086e+02  1.1e+02 -2.0e+01 -3.1e+01 -3.8e+01  3.6e+02 -9.2e-01  4.5e-01  5.3e-01 -1.7e-01 -9.6e-03\n",
      "      200 2.19209e+02  1.1e+02 -2.1e+01 -3.3e+01 -3.8e+01  3.6e+02 -3.0e-02  1.5e-02  1.7e-02 -6.0e-03 -2.6e-07\n",
      "      300 2.19207e+02  1.1e+02 -2.1e+01 -3.3e+01 -3.8e+01  3.6e+02 -1.0e-03  5.1e-04  5.7e-04 -2.0e-04 -6.9e-12\n",
      "      400 2.19207e+02  1.1e+02 -2.1e+01 -3.3e+01 -3.8e+01  3.6e+02 -3.4e-05  1.7e-05  1.9e-05 -6.6e-06 -2.7e-13\n",
      "      500 2.19207e+02  1.1e+02 -2.1e+01 -3.3e+01 -3.8e+01  3.6e+02 -1.1e-06  5.6e-07  6.2e-07 -2.2e-07 -2.6e-13\n",
      "      600 2.19207e+02  1.1e+02 -2.1e+01 -3.3e+01 -3.8e+01  3.6e+02 -3.7e-08  1.9e-08  2.1e-08 -7.3e-09 -2.6e-13\n",
      "      700 2.19207e+02  1.1e+02 -2.1e+01 -3.3e+01 -3.8e+01  3.6e+02 -1.2e-09  6.2e-10  6.9e-10 -2.4e-10 -2.6e-13\n",
      "      800 2.19207e+02  1.1e+02 -2.1e+01 -3.3e+01 -3.8e+01  3.6e+02 -4.1e-11  2.1e-11  2.3e-11 -8.1e-12 -2.7e-13\n",
      "      900 2.19207e+02  1.1e+02 -2.1e+01 -3.3e+01 -3.8e+01  3.6e+02 -1.4e-12  7.0e-13  7.6e-13 -2.7e-13 -2.6e-13\n",
      "w,b found by gradient descent: w: [110.56 -21.27 -32.71 -37.97], b: 363.16\n"
     ]
    }
   ],
   "source": [
    "w_norm, b_norm, hist = run_gradient_descent(X_norm, y_train, 1000, 1.0e-1, )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scaled features get very accurate results **much, much faster!**. Notice the gradient of each parameter is tiny by the end of this fairly short run. A learning rate of 0.1 is a good start for regression with normalized features.\n",
    "Let's plot our predictions versus the target values. Note, the prediction is made using the normalized feature while the plot is shown using the original feature values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 45032 characters]\n",
      "text/plain": [
       "<Figure size 864x216 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#predict target using normalized features\n",
    "m = X_norm.shape[0]\n",
    "yp = np.zeros(m)\n",
    "for i in range(m):\n",
    "    yp[i] = np.dot(X_norm[i], w_norm) + b_norm\n",
    "\n",
    "    # plot predictions and targets versus original features    \n",
    "fig,ax=plt.subplots(1,4,figsize=(12, 3),sharey=True)\n",
    "for i in range(len(ax)):\n",
    "    ax[i].scatter(X_train[:,i],y_train, label = 'target')\n",
    "    ax[i].set_xlabel(X_features[i])\n",
    "    ax[i].scatter(X_train[:,i],yp,color=dlc[\"dlorange\"], label = 'predict')\n",
    "ax[0].set_ylabel(\"Price\"); ax[0].legend();\n",
    "fig.suptitle(\"target versus prediction using z-score normalized model\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results look good. A few points to note:\n",
    "- with multiple features, we can no longer have a single plot showing results versus features.\n",
    "- when generating the plot, the normalized features were used. Any predictions using the parameters learned from a normalized training set must also be normalized."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Prediction**\n",
    "The point of generating our model is to use it to predict housing prices that are not in the data set. Let's predict the price of a house with 1200 sqft, 3 bedrooms, 1 floor, 40 years old. Recall, that you must normalize the data with the mean and standard deviation derived when the training data was normalized. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.53  0.43 -0.79  0.06]\n",
      " predicted price of a house with 1200 sqft, 3 bedrooms, 1 floor, 40 years old = $318709\n"
     ]
    }
   ],
   "source": [
    "# First, normalize out example.\n",
    "x_house = np.array([1200, 3, 1, 40])\n",
    "x_house_norm = (x_house - X_mu) / X_sigma\n",
    "print(x_house_norm)\n",
    "x_house_predict = np.dot(x_house_norm, w_norm) + b_norm\n",
    "print(f\" predicted price of a house with 1200 sqft, 3 bedrooms, 1 floor, 40 years old = ${x_house_predict*1000:0.0f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Cost Contours**  \n",
    "<img align=\"left\" src=\"./images/C1_W2_Lab06_contours.PNG\"   style=\"width:240px;\" >Another way to view feature scaling is in terms of the cost contours. When feature scales do not match, the plot of cost versus parameters in a contour plot is asymmetric. \n",
    "\n",
    "In the plot below, the scale of the parameters is matched. The left plot is the cost contour plot of w[0], the square feet versus w[1], the number of bedrooms before normalizing the features. The plot is so asymmetric, the curves completing the contours are not visible. In contrast, when the features are normalized, the cost contour is much more symmetric. The result is that updates to parameters during gradient descent can make equal progress for each parameter. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 152228 characters]\n",
      "text/plain": [
       "<Figure size 864x360 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt_equal_scale(X_train, X_norm, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Congratulations!\n",
    "In this lab you:\n",
    "- utilized the routines for linear regression with multiple features you developed in previous labs\n",
    "- explored the impact of the learning rate  $\\alpha$ on convergence \n",
    "- discovered the value of feature scaling using z-score normalization in speeding convergence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgments\n",
    "The housing data was derived from the [Ames Housing dataset](http://jse.amstat.org/v19n3/decock.pdf) compiled by Dean De Cock for use in data science education."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
