lecture_2_linear_algebra_2.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Basic setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create anaconda environment\n",
    "<br>\n",
    "```bash\n",
    "conda create -n ml python=3.7.4 jupyter\n",
    "```\n",
    "Install fastai library\n",
    "<br>\n",
    "```bash\n",
    "conda install -c pytorch -c fastai fastai\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Special types of matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Identity matrix $I_{n} \\in \\mathbb{R}^{n \\times n}$ is a matrix which does not change other matrix after multiplication. These kinds of matrices contain ones on main diagonal and zero everywhere else \n",
    "$$\\begin{align} I_{n} &= \\begin{pmatrix}\n",
    "           1, 0, \\dots, 0 \\\\\n",
    "           0, 1, \\dots, 0 \\\\\n",
    "           \\vdots \\\\\n",
    "           0, 0, \\dots, 1 \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "<br>\n",
    "or we can define it with property $\\forall a \\in \\mathbb{R}^{1 \\times n}$ holds $aI_{n} = a$ or $\\forall a \\in \\mathbb{R}^{n \\times 1}$ holds $I_{n}a =a$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "I = np.identity(4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "I"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.random.random(size=(4, 5))\n",
    "B = np.random.random(size=(6, 4))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "B"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "I @ A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "B @ I"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inverse matrix of $A \\in \\mathbb{R}^{n \\times n}$, is the matrix $A^{-1} \\in \\mathbb{R}^{n \\times n}$ for which $A^{-1}A = I$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.random.random(size=(8, 8))\n",
    "A"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "invA = np.linalg.inv(A)\n",
    "invA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Ia = invA @ A\n",
    "print(Ia)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.round(Ia)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vector space "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$v, u \\in \\mathbb{R}^{n}$ and for every $\\alpha \\in \\mathbb{R}^{1}$ we have $u + v \\in \\mathbb{R}^{n}$ and $\\alpha u \\in \\mathbb{R}^{n}$\n",
    "<br>\n",
    "So we have a sum and multiplication on scalar with properties:\n",
    "- for every $u, v, w \\in \\mathbb{R}^{n}$: $(u + v) + w = u + (u + w)$\n",
    "- for every $u, v \\in \\mathbb{R}^{n}$: u + v = v + u\n",
    "- there exists $0 \\in \\mathbb{R}^{n}$ such that: $0 + u = u + 0 = u$\n",
    "- for every $u \\in \\mathbb{R}^{n}$ there exists $-u \\in \\mathbb{R}^{n}$ such that: $u + (-u) = (-u) + u = 0$\n",
    "- for every $\\alpha, \\beta \\in \\mathbb{R}^{1}$ and every $u \\in \\mathbb{R}^{n}$: $\\alpha(\\beta u) = (\\alpha \\beta u)$\n",
    "- for every $u \\in \\mathbb{R}^{n}$: $1u=u1=u$\n",
    "- for every $u, v \\in \\mathbb{R}^{n}$ and $\\alpha \\in \\mathbb{R}^{1}$: $\\alpha (u + v) = \\alpha u + \\alpha v$\n",
    "- for every $\\alpha, \\beta \\in \\mathbb{R}^{1}$ and every $u \\in \\mathbb{R}^{n}$: $(\\alpha + \\beta)u = \\alpha u + \\beta v$\n",
    "<br>\n",
    "So we can define $-$ and $:$ operations as well\n",
    "If some structure satisfies such properties it's called vector space"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "u = np.random.random(5)\n",
    "v = np.random.random(5)\n",
    "x = random.random()\n",
    "y = random.random()\n",
    "w = u * x + v * y\n",
    "u, v, x, y, w\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear combination"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $v_1, v_2, \\dots v_n \\in \\mathbb{R}^{n}$ and $\\alpha_{1}, \\alpha_{2} \\dots \\alpha_{n} \\in \\mathbb{R}^{1}$ then a linear combination of this vectors is called the vector $w = \\sum_{i=1}^{n}\\alpha_{i} v_{i}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Vectors $v_1, v_2, \\dots v_n \\in \\mathbb{R}^{n}$ are called linearly independent, if none of them can be a linear combination of others, of for each $i \\in (1 \\dots n)$ there is no such $\\alpha_{1}, \\alpha_{2} \\dots \\alpha_{i-1}, \\alpha_{i+1} \\dots \\alpha_{n_1} \\in \\mathbb{R}^{1}$ such that $u_i = \\sum_{k = 1, k \\neq i}^{n}\\alpha_{k}u_k $"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The maximum amount of linearly independent vectors in vector space is called the dimension of this space\n",
    "<br>\n",
    "Every vector space has  basis, linearly independent vectors $e_1, e_2 \\dots e_n $ such that every other vector from this space can be achieved by the linear combination of this basis $u = \\sum_{i=1}^n\\alpha_{i}e_i$ and $(\\alpha_{1}, \\alpha_{2}, \\dots, \\alpha_{n})$ are called the coordinates of the vector $u$\n",
    "<br>\n",
    "For $\\mathbb{R}^{n}$ basis is $e_1 = (1, 0, \\dots, 0), e_2 = (0, 1, \\dots, 0), \\dots e_i = (0, 0, \\dots, 1, \\dots, 0), e_n = (0, 0, \\dots, 1)$\n",
    "Or for $\\mathbb{R}^{2}$ basis is $e_1=(1, 0), e_2=(0, 1)$ and for $\\in \\mathbb{R}^{3}$: $e_1=(1, 0, 0), e_2 = (0, 1, 0), e_3=(0, 0, 1)$\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Linear maps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Map $f:X \\to Y$ between vector spaces $\\mathbb{U}$ and $\\mathbb{V}$ is called linear (or linear transformation) if for every $u, v \\in \\mathbb(U)$ and every scalar $\\alpha in \\in \\mathbb{R}^{1}$ we have:\n",
    "- $f(u + v) = f(u) + f(v)$\n",
    "- $f(\\alpha u) = \\alpha f(u)$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $e_1, e_2, \\dots, e_n$ be a basis for linear space $\\mathbb{U}$ and $l_1, l_2, \\dots, l_m$ basis for $\\mathbb{V}$\n",
    "then $f(e_i) = a_1{li}_1 + a_{2i}l_2 + \\dots + a_{mi}l_m$ for some $a_1, \\dots, a_m \\in \\in \\mathbb{R}^{1}$\n",
    "for we have the following matrix:\n",
    "$$\\begin{align} T &= \\begin{pmatrix}\n",
    "           a_{11}, a_{12}, \\dots, a_{1n} \\\\\n",
    "           a_{21}, a_{22}, \\dots, a_{2n} \\\\\n",
    "           \\vdots \\\\\n",
    "           a_{m1}, a_{n2}, \\dots, a_{mn} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "<br>\n",
    "which is called the transformation matrix\n",
    "<br>\n",
    "For each $u \\in \\mathbb{U}$ there exists $b_1, b_2, \\dots, b_n \\in \\mathbb{R}^{1}$ such that: $u = b_1e_1 + b_2e_2 + \\dots + b_ne_n$ and for $f(u)$ (from a linear property) we have $$f(u) = b_1f(e_1) + b_2f(e_2) + \\dots + b_nf(e_n)$$\n",
    "then from the property - $f(e_i) = a_{1i}l_1 + a_{2i}l_2 + \\dots + a_{mi}l_m$, we get:\n",
    "$$f(u) = Tb$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Eigenvectors and Eigenvalues"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If for some linear transformation $T:\\mathbb{U} \\to \\mathbb{V}$ with transformation matrix $M$, there exists non-zero vector $u \\in \\mathbb{U}$ and scalar $\\lambda \\in \\mathbb{R}^{1}$ such that $T(u) = Mu = \\lambda u$, this vector is called eigenvector for the transformation $T$ and $\\lambda$ is called eigenvalue"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$Mu = \\lambda u$ then $Mu - \\lambda u = 0$ and \n",
    "$$(M - I\\lambda)u = 0$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This equation has a solution if $|M - I\\lambda|=0$, thus, we can calculate eigenvector and eigenvalue for linear transformation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider eigenvectors $v_1, v_2, \\dots, v_n$ and eigenvalues $\\lambda_{1}, \\lambda_{2}, \\dots, \\lambda_{n}$ from basis $e_1, e_2, \\dots, e_n$ of transformation matrix $A$, let $Q = (v_1, v_2 \\dots, v_n)$ then:\n",
    "$$AQ = (\\lambda_{1}v_1, \\lambda_{2}v_2, \\dots, \\lambda_{n}v_n)$$\n",
    "define:\n",
    "$$\\begin{align} \\Lambda &= \\begin{pmatrix}\n",
    "           \\lambda_{11}, 0, \\dots, 0 \\\\\n",
    "           0, \\lambda_{22}, \\dots, 0 \\\\\n",
    "           \\vdots \\\\\n",
    "           0, 0, \\dots, \\lambda_{nn} \\\\\n",
    "         \\end{pmatrix}\n",
    "  \\end{align}$$\n",
    "<br>\n",
    "$$AQ = Q\\Lambda$$\n",
    "<br>\n",
    "$$AQQ^{-1}=Q\\Lambda Q^{-1}$$\n",
    "<br>\n",
    "$$AI=Q\\Lambda Q^{-1}$$\n",
    "<br>\n",
    "$$A=Q\\Lambda Q_{-1}$$\n",
    "<br>\n",
    "This is called eigendecomposition of matrix $A$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualisation of Eigenvalues and Eigenvectors"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we consider transformation matrix A again, and we take a bunch of vectors (as much as possible) from given vector \n",
    "space where that matrix is doing transformations, we can find some vectors that never change their orientation but maybe they are scaled by some other factor. \n",
    "\n",
    "Each such vector is called Eigenvector of matrix A which direction isn't affected by transformation, but the scale is affected. The scale of that vector will be an eigenvalue of that vector.\n",
    "\n",
    "Each eigenvector has its eigenvalue and these can be multiple because of several axes of transformation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Eigenvalues and Eigenvectors example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/la2/eigenvalues_and_vectors.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Non-Eigenvalues and Eigenvectors example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/la2/non_eigenvalues_and_vectors.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vector (Matrix) norm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$||x||_{p} = \\sqrt[p]{\\sum_{i=1}^{n}x_i^{p}}$$\n",
    "$$||x||_{p} = (\\sum_{i=1}^{n}x_i^{p})^{1/p}$$\n",
    "<br>\n",
    "$L_2$ norm\n",
    "$$||x||_{2} = (\\sum_{i=1}^{n}x_i^{2})^{1/2}$$\n",
    "$L_1$ norm\n",
    "$$||x||_{1} = \\sum_{i=1}^{n}|x_i|$$\n",
    "<br>\n",
    "$$||x||_{\\infty} = max|x_i|$$\n",
    "\n",
    "For matrices Frobenius norm:\n",
    "$$||A||_{F} = (\\sum_{i=1, j=1}^{n, m}a_{ij})^2$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Determinant as a scaling factor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here is a link to the [video](https://www.youtube.com/watch?v=Ip3X9LOh2dk) about determinants"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Given two vectors x,y in space and some transformation matrix A. \n",
    "If we multiply these vectors by a given transformation matrix, we will get transformed vectors. \n",
    "Area value before and after transformation will be changed with exactly the value of the determinant of A."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/la2/determinant_as_scaling_factor.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## SVD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For any $A \\in \\mathbb{R}^{n \\times m} (\\mathbb{C}^{n \\times m})$ there exists decomposition:\n",
    "$$A = U \\Sigma V^{T}$$ \n",
    "where $U \\in \\mathbb{R}^{n \\times n}(\\mathbb{C}^{n \\times n})$ is a square matrix, $\\Sigma \\in \\mathbb{R}^{n \\times m} (\\mathbb{C}^{n \\times m})$ is a diagonal matrix and  $V \\in \\mathbb{R}^{n \\times n} (\\mathbb{C}^{n \\times n})$ is also a square matrix\n",
    "#### Note: We only discuss real valued vectors, matrices and tensors in this course"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](images/la2/svg1.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "short [tutorial](http://web.mit.edu/be.400/www/SVD/Singular_Value_Decomposition.htm) on SVD from MIT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "very interesting medium [blog](https://medium.com/@jonathan_hui/machine-learning-singular-value-decomposition-svd-principal-component-analysis-pca-1d45e885e491) on SVD & PCA "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![title](images/la2/svd_steps.jpeg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "A = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9], [10, 11, 12]])\n",
    "A, A.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "U, S, V_T = np.linalg.svd(A)\n",
    "U, S, V_T, U.shape, S.shape, V_T.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "Sg = np.diag(S)\n",
    "Sg, Sg.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.linalg.norm(A), np.round(U[:, 0] @ U[:, 1].T), np.linalg.norm(U[2, :]), np.linalg.norm(V_T)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Matrix Decompositions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[fastai LA](https://nbviewer.jupyter.org/github/fastai/numerical-linear-algebra/blob/master/nbs/1.%20Why%20are%20we%20here.ipynb#Matrix-Decompositions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[advanced matrix decompositions](https://sites.google.com/site/igorcarron2/matrixfactorizations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[nfm tutorial](https://perso.telecom-paristech.fr/essid/teach/NMF_tutorial_ICME-2014.pdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[topic modeling](https://medium.com/@nixalo/comp-linalg-l2-topic-modeling-with-nmf-svd-78c94330d45f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[background removal using svd](https://medium.com/@siavashmortezavi/fast-randomized-svd-singular-value-decomposition-using-pytorch-and-gpus-46b627511a6d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PCA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Reduction\n",
    "- PCA is the most commonly used to condense the information contained in a large number of original variables into a smaller set of new composite dimensions, with a minimum loss of information."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[Example](https://www.projectrhea.org/rhea/index.php/PCA_Theory_Examples) of using PCA on image compression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mapping of 2D points into 1D. \n",
    "PCA Takes the most optimal 1d axis to save the data information better, reducing memory by a factor of 2. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![SegmentLocal](images/la2/pca_1d.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interpretation\n",
    "- PCA can be used to discover the important features of a large data set. It often reveals previously unsuspected relationships, thereby allowing interpretations that would not ordinarily result.\n",
    "PCA is typically used as an intermediate step in data analysis when the number of input variables is otherwise too large for useful analysis."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[example](https://towardsdatascience.com/visualising-high-dimensional-datasets-using-pca-and-t-sne-in-python-8ef87e7915b) usage of pca (and t-SNE) for data visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[example](https://github.com/aviolante/sas-python-work/blob/master/tSneExampleBlogPost.ipynb) notebook for comparing PCA and t-SNE for visualizing MNIST data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let $X = (x^{1}, x^{2}, \\dots x^{m})$ is our data where $x^{i} = (x_1^{i}, x_2^{i}, \\dots, x_n^{i}) \\in \\mathbb{R}^{n}$ for each $i \\in (1, 2, \\dots, m)$\n",
    "<br>\n",
    "Normalize data with mean $x^{i} = x^{i} - \\frac{1}{m}\\sum_{i=1}^{m}x^{i}$\n",
    "<br>\n",
    "compute the covariance matrix:\n",
    "$$A = \\frac{1}{m} \\sum_{i=1}^{m}(x^{i})(x^{i})^{T}$$\n",
    "<br>\n",
    "take SVD from $X$:\n",
    "$$ A = U\\Sigma V^{T}$$\n",
    "and consider first $k \\leq n$ columns of $U \\in \\mathbb{R}^{n\\times n}$: \n",
    "$$u_1, u_2, \\dots, u_k \\in \\mathbb{R}^{n}$$\n",
    "now consider the matrix $U_{r} = (u_1, u_2, \\dots, u_k)$ and \n",
    "$$z^{i} = U_{r}^{T}x^{i}$$\n",
    "<br>\n",
    "$U_{r}^{T} \\in \\mathbb{R}^{k \\times n}$ and $x^{i} \\in \\mathbb{R}^{n \\times 1}$ thus $z^{i} \\in \\mathbb{R}^{k \\times 1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reconstruction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can approximate the reconstruction of the original value of $x^{i}$ as \n",
    "$$x_{a}^{i} = U_{r}z^{i}$$\n",
    "<br>\n",
    "$z^{i} \\in \\mathbb{R}^{n \\times 1}$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "to check our method we should compare original value to approximation:\n",
    "$$\\frac{\n",
    "\\frac{1}{m}\\sum_{i=1}^{m}||x^{i} - x_{a}^{i}||^{2}\n",
    "}{\n",
    "\\frac{1}{m}\\sum_{i=1}^{m}||x^{i}||^{2}\n",
    "} \\leq \\epsilon$$\n",
    "<br>\n",
    "$\\epsilon$ might be any value, e.g $\\epsilon = 0.01$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$\\frac{\n",
    "\\frac{1}{m}\\sum_{i=1}^{m}||x^{i} - x_{a}^{i}||^{2}\n",
    "}{\n",
    "\\frac{1}{m}\\sum_{i=1}^{m}||x^{i}||^{2}\n",
    "} \\leq  = 1 -\n",
    "\\frac{\n",
    "\\sum_{i=1}^{k}S_{ii}\n",
    "}{\n",
    "\\sum_{j=1}^{n}S_{jj}\n",
    "}$$\n",
    "<br>\n",
    "So we can calculate\n",
    "$$\\frac{\n",
    "\\sum_{i=1}^{k}S_{ii}\n",
    "}{\n",
    "\\sum_{j=1}^{n}S_{jj}\n",
    "} \\geq \\epsilon$$\n",
    "<br>\n",
    "Only one decomposition is enough"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[pca](https://www.coursera.org/learn/machine-learning/lecture/GBFTt/principal-component-analysis-problem-formulation)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Additional Materials"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[jupyter-notebook tips&tricks&shortcuts](https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}