{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$\\qquad$ $\\qquad$$\\qquad$  **TDA 231 Machine Learning: Homework 4** <br />\n",
    "$\\qquad$ $\\qquad$$\\qquad$ **Goal: Support Vector Machines**<br />\n",
    "$\\qquad$ $\\qquad$$\\qquad$                   **Grader: Vasileios** <br />\n",
    "$\\qquad$ $\\qquad$$\\qquad$                     **Due Date: 14/5** <br />\n",
    "$\\qquad$ $\\qquad$$\\qquad$                   **Submitted by: Name, Personal no., email** <br />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "General guidelines:\n",
    "* All solutions to theoretical problems, can be submitted as a single file named *report.pdf*. They can also be submitted in this ipynb notebook, but equations wherever required, should be formatted using LaTeX math-mode.\n",
    "* All discussion regarding practical problems, along with solutions and plots should be specified here itself. We will not generate the solutions/plots again by running your code.\n",
    "* Your name, personal number and email address should be specified above and also in your file *report.pdf*.\n",
    "* All datasets can be downloaded from the course website.\n",
    "* All tables and other additional information should be included."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Theoretical problems\n",
    "\n",
    "## [SVM, 5 points]\n",
    "\n",
    "Consider a (hard margin) SVM with the following training points from\n",
    "two classes:\n",
    "\\begin{eqnarray}\n",
    "+1: &(2,2), (4,4), (4,0) \\nonumber \\\\\n",
    "-1: &(0,0), (2,0), (0,2) \\nonumber\n",
    "\\end{eqnarray}\n",
    "\n",
    "Plot these six training points, and construct by inspection the\n",
    "weight vector for the optimal hyperplane. In your solution, specify\n",
    "the hyperplane in terms of w and b such that $w_1 x_1 + w_2 x_2 + b =\n",
    "0$. Calculate what the margin is (i.e., $2\\gamma$ where $\\gamma$ is the\n",
    "distance from the hyperplane to its closest data point), showing all\n",
    "of your work. (Hint: It may be useful to recall that the distance of a point $(a_1,a_2)$ from the line $w_1x_1 + w_2x_2 + b = 0$ is $|w_1a_1 + w_2a_2 + b|/\\sqrt{w_1^2 + w_2^2}$.)\n",
    "\n",
    "## [SVM cont'd, 5 points]\n",
    "\n",
    "\n",
    "Consider the same problem from above.\n",
    "\n",
    "a. Write the primal formulation of the SVM **for this specific example** i.e. you have to specialise the general formulation for the set of inputs given.\n",
    "\n",
    "b. Give the optimal primal solution **for this specific**.\n",
    "\n",
    "c. Write the dual formulation **for this specific**.\n",
    "\n",
    "d. Give the optimal dual solution, comment on support vectors."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical problems\n",
    "\n",
    "All data files can be loaded using the following code for example:\n",
    "```python\n",
    "import numpy as np\n",
    "data = np.loadtxt('d1.txt')\n",
    "X = data[:,:2]\n",
    "Y = data[:,-1]\n",
    "```\n",
    "\n",
    "For this assigment, you may use sklearn.svm.SVC or sklearn.svm.NuSVC. \n",
    "\n",
    "## [SVM, 5 points]\n",
    "\n",
    "Consider the dataset **d1.txt** having two-dimensional points $X$ and corresponding labels $Y$.\n",
    "\n",
    "a. Train an SVM (with linear kernel) on the above data.\n",
    "\n",
    "b. Plot the data and the separating hyperplane of the trained classifier. Color the points of one class red and the other blue. Mark points that could not be separated correctly. Also, mark the support vectors.\n",
    "\n",
    "c. Does the classifier have bias, and if so, what is it?\n",
    "\n",
    "d. What is the (soft) margin?\n",
    "\n",
    "## [Kernels, 5 points]\n",
    "\n",
    "Download the dataset **d2.txt**. Consider the following kernels.\n",
    "\n",
    "* Linear kernel\n",
    "* Quadratic kernel \n",
    "* Radial Basis Function (RBF) kernel\n",
    "\n",
    "a. Train an SVM with the linear kernel on all of the data (d2.txt). Plot the data. Color the points of one class red and the other blue. Mark points that could not be separated correctly by the trained model.\n",
    "\n",
    "b. Plot the decision surfaces for the Quadratic and RBF kernel seperately."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}