{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Optional Lab: Feature Engineering and Polynomial Regression\n",
    "\n",
    "![](./images/C1_W2_Lab07_FeatureEngLecture.PNG)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Goals\n",
    "In this lab you will:\n",
    "- explore feature engineering and polynomial regression which allows you to use the machinery of linear regression to fit very complicated, even very non-linear functions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tools\n",
    "You will utilize the function developed in previous labs as well as matplotlib and NumPy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "from lab_utils_multi import zscore_normalize_features, run_gradient_descent_feng\n",
    "np.set_printoptions(precision=2)  # reduced display precision on numpy arrays"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='FeatureEng'></a>\n",
    "# Feature Engineering and Polynomial Regression Overview\n",
    "\n",
    "Out of the box, linear regression provides a means of building models of the form:\n",
    "$$f_{\\mathbf{w},b} = w_0x_0 + w_1x_1+ ... + w_{n-1}x_{n-1} + b \\tag{1}$$ \n",
    "What if your features/data are non-linear or are combinations of features? For example,  Housing prices do not tend to be linear with living area but penalize very small or very large houses resulting in the curves shown in the graphic above. How can we use the machinery of linear regression to fit this curve? Recall, the 'machinery' we have is the ability to modify the parameters $\\mathbf{w}$, $\\mathbf{b}$ in (1) to 'fit' the equation to the training data. However, no amount of adjusting of $\\mathbf{w}$,$\\mathbf{b}$ in (1) will achieve a fit to a non-linear curve.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a name='PolynomialFeatures'></a>\n",
    "## Polynomial Features\n",
    "\n",
    "Above we were considering a scenario where the data was non-linear. Let's try using what we know so far to fit a non-linear curve. We'll start with a simple quadratic: $y = 1+x^2$\n",
    "\n",
    "You're familiar with all the routines we're using. They are available in the lab_utils.py file for review. We'll use [`np.c_[..]`](https://numpy.org/doc/stable/reference/generated/numpy.c_.html) which is a NumPy routine to concatenate along the column boundary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration         0, Cost: 1.65756e+03\n",
      "Iteration       100, Cost: 6.94549e+02\n",
      "Iteration       200, Cost: 5.88475e+02\n",
      "Iteration       300, Cost: 5.26414e+02\n",
      "Iteration       400, Cost: 4.90103e+02\n",
      "Iteration       500, Cost: 4.68858e+02\n",
      "Iteration       600, Cost: 4.56428e+02\n",
      "Iteration       700, Cost: 4.49155e+02\n",
      "Iteration       800, Cost: 4.44900e+02\n",
      "Iteration       900, Cost: 4.42411e+02\n",
      "w,b found by gradient descent: w: [18.7], b: -52.0834\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 22072 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# create target data\n",
    "x = np.arange(0, 20, 1)\n",
    "y = 1 + x**2\n",
    "X = x.reshape(-1, 1)\n",
    "\n",
    "model_w,model_b = run_gradient_descent_feng(X,y,iterations=1000, alpha = 1e-2)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"no feature engineering\")\n",
    "plt.plot(x,X@model_w + model_b, label=\"Predicted Value\");  plt.xlabel(\"X\"); plt.ylabel(\"y\"); plt.legend(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, as expected, not a great fit. What is needed is something like $y= w_0x_0^2 + b$, or a **polynomial feature**.\n",
    "To accomplish this, you can modify the *input data* to *engineer* the needed features. If you swap the original data with a version that squares the $x$ value, then you can achieve $y= w_0x_0^2 + b$. Let's try it. Swap `X` for `X**2` below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create target data\n",
    "x = np.arange(0, 20, 1)\n",
    "y = 1 + x**2\n",
    "\n",
    "# Engineer features \n",
    "X = x**2      #<-- added engineered feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration         0, Cost: 7.32922e+03\n",
      "Iteration      1000, Cost: 2.24844e-01\n",
      "Iteration      2000, Cost: 2.22795e-01\n",
      "Iteration      3000, Cost: 2.20764e-01\n",
      "Iteration      4000, Cost: 2.18752e-01\n",
      "Iteration      5000, Cost: 2.16758e-01\n",
      "Iteration      6000, Cost: 2.14782e-01\n",
      "Iteration      7000, Cost: 2.12824e-01\n",
      "Iteration      8000, Cost: 2.10884e-01\n",
      "Iteration      9000, Cost: 2.08962e-01\n",
      "w,b found by gradient descent: w: [1.], b: 0.0490\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 25380 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "X = X.reshape(-1, 1)  #X should be a 2-D Matrix\n",
    "model_w,model_b = run_gradient_descent_feng(X, y, iterations=10000, alpha = 1e-5)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"Added x**2 feature\")\n",
    "plt.plot(x, np.dot(X,model_w) + model_b, label=\"Predicted Value\"); plt.xlabel(\"x\"); plt.ylabel(\"y\"); plt.legend(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! near perfect fit. Notice the values of $\\mathbf{w}$ and b printed right above the graph: `w,b found by gradient descent: w: [1.], b: 0.0490`. Gradient descent modified our initial values of $\\mathbf{w},b $ to be (1.0,0.049) or a model of $y=1*x_0^2+0.049$, very close to our target of $y=1*x_0^2+1$. If you ran it longer, it could be a better match. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting Features\n",
    "<a name='GDF'></a>\n",
    "Above, we knew that an $x^2$ term was required. It may not always be obvious which features are required. One could add a variety of potential features to try and find the most useful. For example, what if we had instead tried : $y=w_0x_0 + w_1x_1^2 + w_2x_2^3+b$ ? \n",
    "\n",
    "Run the next cells. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create target data\n",
    "x = np.arange(0, 20, 1)\n",
    "y = x**2\n",
    "\n",
    "# engineer features .\n",
    "X = np.c_[x, x**2, x**3]   #<-- added engineered feature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration         0, Cost: 1.14029e+03\n",
      "Iteration      1000, Cost: 3.28539e+02\n",
      "Iteration      2000, Cost: 2.80443e+02\n",
      "Iteration      3000, Cost: 2.39389e+02\n",
      "Iteration      4000, Cost: 2.04344e+02\n",
      "Iteration      5000, Cost: 1.74430e+02\n",
      "Iteration      6000, Cost: 1.48896e+02\n",
      "Iteration      7000, Cost: 1.27100e+02\n",
      "Iteration      8000, Cost: 1.08495e+02\n",
      "Iteration      9000, Cost: 9.26132e+01\n",
      "w,b found by gradient descent: w: [0.08 0.54 0.03], b: 0.0106\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 26064 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "model_w,model_b = run_gradient_descent_feng(X, y, iterations=10000, alpha=1e-7)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"x, x**2, x**3 features\")\n",
    "plt.plot(x, X@model_w + model_b, label=\"Predicted Value\"); plt.xlabel(\"x\"); plt.ylabel(\"y\"); plt.legend(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note the value of $\\mathbf{w}$, `[0.08 0.54 0.03]` and b is `0.0106`.This implies the model after fitting/training is:\n",
    "$$ 0.08x + 0.54x^2 + 0.03x^3 + 0.0106 $$\n",
    "Gradient descent has emphasized the data that is the best fit to the $x^2$ data by increasing the $w_1$ term relative to the others.  If you were to run for a very long time, it would continue to reduce the impact of the other terms. \n",
    ">Gradient descent is picking the 'correct' features for us by emphasizing its associated parameter\n",
    "\n",
    "Let's review this idea:\n",
    "- Intially, the features were re-scaled so they are comparable to each other\n",
    "- less weight value implies less important/correct feature, and in extreme, when the weight becomes zero or very close to zero, the associated feature is not useful in fitting the model to the data.\n",
    "- above, after fitting, the weight associated with the $x^2$ feature is much larger than the weights for $x$ or $x^3$ as it is the most useful in fitting the data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### An Alternate View\n",
    "Above, polynomial features were chosen based on how well they matched the target data. Another way to think about this is to note that we are still using linear regression once we have created new features. Given that, the best features will be linear relative to the target. This is best understood with an example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create target data\n",
    "x = np.arange(0, 20, 1)\n",
    "y = x**2\n",
    "\n",
    "# engineer features .\n",
    "X = np.c_[x, x**2, x**3]   #<-- added engineered feature\n",
    "X_features = ['x','x^2','x^3']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 11648 characters]\n",
      "text/plain": [
       "<Figure size 864x216 with 3 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig,ax=plt.subplots(1, 3, figsize=(12, 3), sharey=True)\n",
    "for i in range(len(ax)):\n",
    "    ax[i].scatter(X[:,i],y)\n",
    "    ax[i].set_xlabel(X_features[i])\n",
    "ax[0].set_ylabel(\"y\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, it is clear that the $x^2$ feature mapped against the target value $y$ is linear. Linear regression can then easily generate a model using that feature."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scaling features\n",
    "As described in the last lab, if the data set has features with significantly different scales, one should apply feature scaling to speed gradient descent. In the example above, there is $x$, $x^2$ and $x^3$ which will naturally have very different scales. Let's apply Z-score normalization to our example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Peak to Peak range by column in Raw        X:[  19  361 6859]\n",
      "Peak to Peak range by column in Normalized X:[3.3  3.18 3.28]\n"
     ]
    }
   ],
   "source": [
    "# create target data\n",
    "x = np.arange(0,20,1)\n",
    "X = np.c_[x, x**2, x**3]\n",
    "print(f\"Peak to Peak range by column in Raw        X:{np.ptp(X,axis=0)}\")\n",
    "\n",
    "# add mean_normalization \n",
    "X = zscore_normalize_features(X)     \n",
    "print(f\"Peak to Peak range by column in Normalized X:{np.ptp(X,axis=0)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can try again with a more aggressive value of alpha:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration         0, Cost: 9.42147e+03\n",
      "Iteration     10000, Cost: 3.90938e-01\n",
      "Iteration     20000, Cost: 2.78389e-02\n",
      "Iteration     30000, Cost: 1.98242e-03\n",
      "Iteration     40000, Cost: 1.41169e-04\n",
      "Iteration     50000, Cost: 1.00527e-05\n",
      "Iteration     60000, Cost: 7.15855e-07\n",
      "Iteration     70000, Cost: 5.09763e-08\n",
      "Iteration     80000, Cost: 3.63004e-09\n",
      "Iteration     90000, Cost: 2.58497e-10\n",
      "w,b found by gradient descent: w: [5.27e-05 1.13e+02 8.43e-05], b: 123.5000\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 27820 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(0,20,1)\n",
    "y = x**2\n",
    "\n",
    "X = np.c_[x, x**2, x**3]\n",
    "X = zscore_normalize_features(X) \n",
    "\n",
    "model_w, model_b = run_gradient_descent_feng(X, y, iterations=100000, alpha=1e-1)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"Normalized x x**2, x**3 feature\")\n",
    "plt.plot(x,X@model_w + model_b, label=\"Predicted Value\"); plt.xlabel(\"x\"); plt.ylabel(\"y\"); plt.legend(); plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feature scaling allows this to converge much faster.   \n",
    "Note again the values of $\\mathbf{w}$. The $w_1$ term, which is the $x^2$ term is the most emphasized. Gradient descent has all but eliminated the $x^3$ term."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Complex Functions\n",
    "With feature engineering, even quite complex functions can be modeled:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Iteration         0, Cost: 2.20188e-01\n",
      "Iteration    100000, Cost: 1.70074e-02\n",
      "Iteration    200000, Cost: 1.27603e-02\n",
      "Iteration    300000, Cost: 9.73032e-03\n",
      "Iteration    400000, Cost: 7.56440e-03\n",
      "Iteration    500000, Cost: 6.01412e-03\n",
      "Iteration    600000, Cost: 4.90251e-03\n",
      "Iteration    700000, Cost: 4.10351e-03\n",
      "Iteration    800000, Cost: 3.52730e-03\n",
      "Iteration    900000, Cost: 3.10989e-03\n",
      "w,b found by gradient descent: w: [ -1.34 -10.    24.78   5.96 -12.49 -16.26  -9.51   0.59   8.7   11.94\n",
      "   9.27   0.79 -12.82], b: -0.0073\n"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 30316 characters]\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x = np.arange(0,20,1)\n",
    "y = np.cos(x/2)\n",
    "\n",
    "X = np.c_[x, x**2, x**3,x**4, x**5, x**6, x**7, x**8, x**9, x**10, x**11, x**12, x**13]\n",
    "X = zscore_normalize_features(X) \n",
    "\n",
    "model_w,model_b = run_gradient_descent_feng(X, y, iterations=1000000, alpha = 1e-1)\n",
    "\n",
    "plt.scatter(x, y, marker='x', c='r', label=\"Actual Value\"); plt.title(\"Normalized x x**2, x**3 feature\")\n",
    "plt.plot(x,X@model_w + model_b, label=\"Predicted Value\"); plt.xlabel(\"x\"); plt.ylabel(\"y\"); plt.legend(); plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Congratulations!\n",
    "In this lab you:\n",
    "- learned how linear regression can model complex, even highly non-linear functions using feature engineering\n",
    "- recognized that it is important to apply feature scaling when doing feature engineering"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "toc-autonumbering": false
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
