Linear_Algebra_with_Python.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Linear Algebra with Python.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "am9Zx3-uK3Su"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/reddyprasade/Numpy-with-Python/blob/master/Linear_Algebra_with_Python.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u9qQJ3JIK3Ru",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "## Python and Pip\n",
        "\n",
        "If you are using your own PC, Anaconda is recommended. But at least, please download [Python](https://www.python.org/downloads/) and [Pip](https://pip.pypa.io/en/stable/installing/). \n",
        "\n",
        "\n",
        "## Libraries\n",
        "\n",
        "We'll be working with `numpy` and `scipy`, so make sure to install them. Pull up your terminal and insert the following: \n",
        "\n",
        "```\n",
        "pip3 install scipy\n",
        "pip3 install numpy\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DJ6ZZORLK3R6",
        "colab_type": "text"
      },
      "source": [
        "## Vectors\n",
        "\n",
        "A vector is typically an ordered tuple of numbers which have both a magnitude and direction. r. \n",
        "\n",
        "With that said, we <i>can</i> represent a vector with a list, for example: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3fNcTE70K3R7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "A = [2.0, 3.0, 5.0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ZDnGBC0_K3SA",
        "colab_type": "text"
      },
      "source": [
        "Now, let's write this same vector with `numpy` instead:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GNzznz7mK3SB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "v = np.array([2.0, 3.0, 5.0])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WElV1yngK3SP",
        "colab_type": "text"
      },
      "source": [
        "### What is a Norm? \n",
        "\n",
        "A norm just refers to the magnitude of a vector, and is denoted with ||u||. With numpy and scipy, we can do calculate the norm as follows: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ldaN-dLIK3SQ",
        "colab_type": "code",
        "outputId": "1180646c-f69e-4843-f2b5-fadf561ff5fa",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "v = np.array([1,2,3,4])\n",
        "nln.norm(v)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "5.4772255750516612"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5XgY4rEBK3SY",
        "colab_type": "text"
      },
      "source": [
        "The actual formula looks like: \n",
        "\n",
        "$$ ||v|| = \\sqrt{v_1^2 + ... + v_n^2} $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XSxqgO7zK3Sf",
        "colab_type": "text"
      },
      "source": [
        "## Matrices\n",
        "\n",
        "A Matrix is a 2D array that stores real or complex numbers.  You can use numpy to create matrices:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CdZt9To6K3Sh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "matrix1 = np.matrix(\n",
        "    [[0, 4],\n",
        "     [2, 0]]\n",
        ")\n",
        "matrix2 = np.matrix(\n",
        "    [[-1, 2],\n",
        "     [1, -2]]\n",
        ")"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "DxVHpcOWK3St",
        "colab_type": "text"
      },
      "source": [
        "# 1.0 Linear Equations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "am9Zx3-uK3Su",
        "colab_type": "text"
      },
      "source": [
        "## Gauss-Jordan Elimination\n",
        "\n",
        "Gaussian Elimination helps to put a matrix in row echelon form, while Gauss-Jordan Elimination puts a matrix in reduced row echelon form?"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oFcw7VdZK3Sv",
        "colab_type": "text"
      },
      "source": [
        "## Solving Systems of Equations\n",
        "\n",
        "Consider a set of m linear equations in n unknowns:\n",
        "\n",
        "...\n",
        "\n",
        "We can let:\n",
        "\n",
        "...\n",
        "\n",
        "And re-write the system: \n",
        "\n",
        "```\n",
        "Ax = b\n",
        "```\n",
        "This reduces the problem to a matrix equation and now we can solve the system to find $A^{-1}$.\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "klVt4KBKK3Sw",
        "colab_type": "text"
      },
      "source": [
        "### Systems of Equations with Python\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PncWhhgaK3Sx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "A = np.array([ [3,-9], [2,4] ])\n",
        "b = np.array([-42,2])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TVCw47hSK3S5",
        "colab_type": "text"
      },
      "source": [
        "Now, we can use the `linalg.solve()` function to solve the x and y values. Note that these values will be\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "J1fylfX0K3S7",
        "colab_type": "code",
        "outputId": "402d4d20-2c40-42a1-8c37-324a8bd8d176",
        "colab": {}
      },
      "source": [
        "np.linalg.solve(A,b)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([-5.,  3.])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "frzIinyAK3TC",
        "colab_type": "text"
      },
      "source": [
        "# 2.0 Linear Transformations"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mU2Nc6R2K3TF",
        "colab_type": "text"
      },
      "source": [
        "## Matrix Operations\n",
        "\n",
        "What makes matrices particularly useful is the fact that we can perform operations on them. While it won't be necessarily intuitive why these operations are important right now, it will become obvious in later content.\n",
        "\n",
        "### Addition\n",
        "\n",
        "Matrix addition works very similarlty to normal addition. You simply add the corresponding spots together. \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "o_fndND0K3TG",
        "colab_type": "code",
        "outputId": "f47fb293-9891-4790-d116-d37c26e0135e",
        "colab": {}
      },
      "source": [
        "matrix1 + matrix2"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "matrix([[-1,  6],\n",
              "        [ 3, -2]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 82
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zziOPjBWK3TK",
        "colab_type": "text"
      },
      "source": [
        "### Multiplication\n",
        "\n",
        "To multiply two matrices with numpy, you can use the `np.dot` method: \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z1aINdHSK3TL",
        "colab_type": "code",
        "outputId": "f2dd8f37-fb78-42d7-f114-f676b031e256",
        "colab": {}
      },
      "source": [
        "np.dot(matrix1, matrix2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "matrix([[ 4, -8],\n",
              "        [-2,  4]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 71
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wBZ8XZDQK3TP",
        "colab_type": "text"
      },
      "source": [
        "Or, simply, you can do:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "KU1AceM6K3TQ",
        "colab_type": "code",
        "outputId": "a2af2b9c-fa03-47bf-8002-107d811627bd",
        "colab": {}
      },
      "source": [
        "matrix1 * matrix2"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "matrix([[ 4, -8],\n",
              "        [-2,  4]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lsWBeJfuK3TU",
        "colab_type": "text"
      },
      "source": [
        "The dot product is an operation that takes two coordinate vectors of equal size and returns a single number. The result is calculated by multiplying corresponding entries and adding up those products. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "macwSed5K3TV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "v5W4nH9SK3TY",
        "colab_type": "text"
      },
      "source": [
        "### Identity Matrix\n",
        "\n",
        "A Diagonal Matrix is an n x n matrix with 1s on the diagonal from the top left to the bottom right, such as \n",
        "\n",
        "``` \n",
        "[[ 1., 0., 0.],\n",
        "[ 0., 1., 0.],\n",
        "[ 0., 0., 1.]]\n",
        "```\n",
        "\n",
        "We can generate diagonal matrices with the `eye()` function in Python: \n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "78lISXDNK3TZ",
        "colab_type": "code",
        "outputId": "1a7097c4-96c1-481f-b78a-0e9aea175e07",
        "colab": {}
      },
      "source": [
        "np.eye(4)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[ 1.,  0.,  0.,  0.],\n",
              "       [ 0.,  1.,  0.,  0.],\n",
              "       [ 0.,  0.,  1.,  0.],\n",
              "       [ 0.,  0.,  0.,  1.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 80
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hSD7hE-3K3Tb",
        "colab_type": "text"
      },
      "source": [
        "When a matrix is multiplied by its inverse, the result is the identity matrix. It's important to note that only square matrices have inverses!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "rF7u7EjWK3Td",
        "colab_type": "text"
      },
      "source": [
        "### Inverse Matrices\n",
        "\n",
        "The matrix A is invertible if there exists a matrix A<sub>-1</sub> such that\n",
        "\n",
        "A<sub>-1</sub>A = I and AA<sub>-1</sub> = I\n",
        "\n",
        "Multiplying inverse matrix is like the division; itâ€™s simply the reciprocal function of multiplication. Letâ€™s say here is a square matrix A, then multiplying its inversion gives the identity matrix I.\n",
        "\n",
        "We can get the inverse matrix with numpy: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2mnKTVs7K3Tf",
        "colab_type": "code",
        "outputId": "c5410df7-5b78-4c09-8dc0-3bab9ada5ac1",
        "colab": {}
      },
      "source": [
        "inverse = np.linalg.inv(matrix1)\n",
        "print(inverse)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[[ 0.    0.5 ]\n",
            " [ 0.25  0.  ]]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JWPWoId-K3Tt",
        "colab_type": "text"
      },
      "source": [
        "# 3.0 Orthogonality and Least Squares\n",
        "\n",
        "\n",
        "## QR Decomposition with Gram-Schmidt\n",
        "\n",
        "\n",
        "QR decomposition (also called a QR factorization) of a matrix is a decomposition of a matrix A into a product A = QR of an orthogonal matrix Q and an upper triangular matrix R. QR decomposition is often used to solve the linear least squares problem and is the basis for a particular eigenvalue algorithm, the QR algorithm."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sbOpDGUZK3Tt",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "A = np.matrix([[1,1,0], [1,0,1], [0,1,1]])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "deznRWKSK3Tv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "Q,R = np.linalg.qr(A)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "-bU4A2gDK3Tx",
        "colab_type": "code",
        "outputId": "61e36254-df27-4ec7-eafa-bee78a465434",
        "colab": {}
      },
      "source": [
        "Q"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "matrix([[-0.70710678,  0.40824829, -0.57735027],\n",
              "        [-0.70710678, -0.40824829,  0.57735027],\n",
              "        [-0.        ,  0.81649658,  0.57735027]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 75
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EewOUX4nK3T0",
        "colab_type": "code",
        "outputId": "7acee8df-af04-4ac0-da5e-3039aaa3b5b5",
        "colab": {}
      },
      "source": [
        "R"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "matrix([[-1.41421356, -0.70710678, -0.70710678],\n",
              "        [ 0.        ,  1.22474487,  0.40824829],\n",
              "        [ 0.        ,  0.        ,  1.15470054]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 76
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WdQWfrqEK3T3",
        "colab_type": "text"
      },
      "source": [
        "# 4.0 Determinants"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5S9ARsHdK3T5",
        "colab_type": "text"
      },
      "source": [
        "### Trace and Determinant\n",
        "\n",
        "The trace of a matrix A is the sum of its diagonal elements. It's important because it's an invariant of a matrix under change of basis and it defines a matrix norm. \n",
        "\n",
        "In Python, this can be done with the `numpy` module: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y5M80gd6K3T5",
        "colab_type": "code",
        "outputId": "ec982f55-a8f5-4580-8f8a-08a265399b49",
        "colab": {}
      },
      "source": [
        "print(np.trace(matrix1))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WJYQVjiWK3T8",
        "colab_type": "text"
      },
      "source": [
        "which in this case is just `0`. \n",
        "\n",
        "The determinant of a matrix is defined to be the alternating sum of permutations of the elements of a matrix. The formula is as follows:"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vhN6rqSzK3T9",
        "colab_type": "text"
      },
      "source": [
        "\\begin{bmatrix} \n",
        "    a      & b  \\\\\n",
        "    c      & d  \\\\\n",
        "\\end{bmatrix}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ig8shVDKK3T-",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "$$ = ad - bc $$"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bTUTIbSoK3UD",
        "colab_type": "text"
      },
      "source": [
        "In python, you can use the following function:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "59Cbf6Q4K3UE",
        "colab_type": "code",
        "outputId": "fac68b2f-393f-40ab-cb97-1e7ead3bf172",
        "colab": {}
      },
      "source": [
        "det = np.linalg.det(matrix1)\n",
        "print(det)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "-8.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ou7UPGR0K3UI",
        "colab_type": "text"
      },
      "source": [
        "Note that an n×n matrix A is invertible $\\iff$ det(A) $\\ne$ 0."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3X48rbZ5K3UO",
        "colab_type": "text"
      },
      "source": [
        "## Matrix Types\n",
        "\n",
        "\n",
        "\n",
        "### Transpose\n",
        "\n",
        "The transpose of a matrix is another operation that can be accomplished with `numpy`. Its importance isn't so obvious as first, but they're used to <HERE>. Now, let's take a look at an example. Given the matrix below, let's find what its transpose looks like. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gi9ide7tK3UQ",
        "colab_type": "code",
        "outputId": "66fb0970-58ad-4a78-b87b-e6fd166375fe",
        "colab": {}
      },
      "source": [
        "a = np.array([[1, 2], [3, 4]])\n",
        "a"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1, 2],\n",
              "       [3, 4]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 83
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "FoXPIIUzK3US",
        "colab_type": "text"
      },
      "source": [
        "Notice the the 2 and 3 flipped positions. The transpose essentially flips the positions of a matrix in a particular way. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xlsz9SlOK3UT",
        "colab_type": "code",
        "outputId": "d7e8a92a-9e75-49e9-e3c7-616f4af3ce59",
        "colab": {}
      },
      "source": [
        "a.transpose()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1, 3],\n",
              "       [2, 4]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 84
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PSEfGCAhK3UX",
        "colab_type": "text"
      },
      "source": [
        "So what does that look like when we have a matrix that's not 2x2 dimensions? "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "iwUGams3K3UX",
        "colab_type": "code",
        "outputId": "bdeb9f8c-eb9e-4b5c-ad9b-7c4728aa8dfe",
        "colab": {}
      },
      "source": [
        "a = np.array([[1, 2, 3], [4, 5, 6]])\n",
        "a"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1, 2, 3],\n",
              "       [4, 5, 6]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 85
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m0U3kUjeK3Ua",
        "colab_type": "text"
      },
      "source": [
        "The number of dimensions is now flipped as well. Instead of three columns, we now have two. And instead of two rows, we now have three. "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OlKB548kK3Ua",
        "colab_type": "code",
        "outputId": "922bb3c0-330f-44c0-ae60-63093bbfdd78",
        "colab": {}
      },
      "source": [
        "a.transpose()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([[1, 4],\n",
              "       [2, 5],\n",
              "       [3, 6]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 86
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d6YcmblZK3Uh",
        "colab_type": "text"
      },
      "source": [
        "# 5.0 Eigenvalues & Eigenvectors\n",
        "\n",
        "Let A be an `n x n` matrix. The number &lambda; is an eigenvalue of `A` if there exists a non-zero vector `C` such that\n",
        "\n",
        "Av = &lambda;v\n",
        "\n",
        "In this case, vector `v` is called an eigenvector of `A` corresponding to &lambda;. You can use numpy to calculate the eigenvalues and eigenvectors of a matrix: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WdDxo0YlK3Ui",
        "colab_type": "code",
        "outputId": "fdb970df-8f04-4f90-c568-db859dc15631",
        "colab": {}
      },
      "source": [
        "eigenvecs, eigvals = np.linalg.eigvals(matrix1)\n",
        "print(eigenvecs, eigvals)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2.82842712475 -2.82842712475\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LyGx6bS8K3Un",
        "colab_type": "text"
      },
      "source": [
        "It's important to note that eigenvectors do not change direction in the transformation of the matrix.\n"
      ]
    }
  ]
}