08_Numerical_stability_with_PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2009161a-5bc2-4b4c-a512-77dd7783d9d3",
   "metadata": {},
   "source": [
    "Numerical stability refers to the ability of a numerical algorithm to produce reliable and accurate results even in the presence of small perturbations or rounding errors. In PyTorch, as in any numerical computing framework, ensuring numerical stability is crucial, especially when dealing with deep learning models that involve complex computations with large numbers of parameters.\n",
    "\n",
    "When using any numerical computation library such as NumPy or PyTorch, it's important to note that writing mathematically correct code doesn't necessarily lead to correct results. You also need to make sure that the computations are stable.\n",
    "\n",
    "Here are some key considerations for achieving numerical stability in PyTorch:\n",
    "\n",
    "1. **Numerically Stable Operations**: Certain mathematical operations, such as addition, subtraction, multiplication, and division, can lead to numerical instability, especially when dealing with very large or very small numbers. PyTorch provides functions like torch.add(), torch.sub(), torch.mul(), and torch.div() that are implemented to handle numerical stability.\n",
    "\n",
    "2. **Avoiding Underflow and Overflow**: Underflow occurs when numbers close to zero become rounded to zero, while overflow occurs when numbers with large magnitudes cannot be represented accurately. To mitigate these issues, it's important to scale the input data appropriately, use proper initialization techniques for model parameters, and employ techniques like gradient clipping to prevent exploding gradients.\n",
    "\n",
    "3. **Numerically Stable Loss Functions**: Certain loss functions, such as softmax cross-entropy or log-likelihood loss, can become numerically unstable, particularly when dealing with probabilities or log probabilities. PyTorch provides implementations of these loss functions (torch.nn.CrossEntropyLoss, torch.nn.NLLLoss, etc.) that handle numerical stability internally.\n",
    "\n",
    "4. **Gradient Scaling**: During training, gradient values can become very large or very small, leading to numerical instability. Gradient scaling techniques, such as gradient clipping or gradient normalization, help prevent gradients from becoming too large or too small, thereby improving numerical stability.\n",
    "\n",
    "5. **Floating-point Precision**: PyTorch supports different floating-point precisions (e.g., float32, float64) for numerical computations. Using higher precision (e.g., float64) may provide increased numerical stability but comes with a trade-off in terms of memory consumption and computation speed.\n",
    "\n",
    "6. **Batch Normalization and Regularization**: Techniques like batch normalization and regularization (e.g., L1 or L2 regularization) can also help improve numerical stability by preventing activations from drifting too far from zero or by constraining the magnitude of the model parameters.\n",
    "\n",
    "7. **Numerically Stable Initialization**: Choosing appropriate initialization schemes for model parameters (e.g., Xavier/Glorot initialization, He initialization) can help improve convergence and stability during training.\n",
    "\n",
    "By paying attention to these considerations and adopting best practices, such as using PyTorch's built-in functions and modules designed for numerical stability, you can ensure that your deep learning models in PyTorch are robust and reliable even in challenging numerical conditions."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "181b05bc-c0e7-40c4-b6e4-62bf68aa3912",
   "metadata": {},
   "source": [
    "#### Example 1: Numerical Instability in Basic Arithmetic Operations (using NumPy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "51c5b2a8-5cda-4ca6-87ce-75e37b24fc51",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nan\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Hp\\AppData\\Local\\Temp\\ipykernel_21064\\612643738.py:8: RuntimeWarning: invalid value encountered in scalar divide\n",
      "  z = x * y / y\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Define values for x and y\n",
    "x = np.float32(1)\n",
    "y = np.float32(1e-50)  # y would be stored as zero\n",
    "\n",
    "# Perform the operation x * y / y\n",
    "z = x * y / y\n",
    "\n",
    "print(z)  # prints nan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0885b473-bbc9-4e67-8371-7bd901c71803",
   "metadata": {},
   "source": [
    "Explanation:\r\n",
    "In this example, y is too small to be accurately represented as a float32. Therefore, it is stored as zero. When dividing x by y, we encounter a division by zero, resulting in nan (not a number)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34ba7e7b-ef5c-4fd8-80ea-36a611a3c81b",
   "metadata": {},
   "source": [
    "#### Example 2: Avoiding Overflow (using NumPy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "28f83320-3d76-40ce-b91c-a6076e2d093d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "nan\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Hp\\AppData\\Local\\Temp\\ipykernel_21064\\2624422379.py:5: RuntimeWarning: overflow encountered in cast\n",
      "  y = np.float32(1e39)  # y would be stored as inf\n",
      "C:\\Users\\Hp\\AppData\\Local\\Temp\\ipykernel_21064\\2624422379.py:8: RuntimeWarning: invalid value encountered in scalar divide\n",
      "  z = x * y / y\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Define values for x and y\n",
    "x = np.float32(1)\n",
    "y = np.float32(1e39)  # y would be stored as inf\n",
    "\n",
    "# Perform the operation x * y / y\n",
    "z = x * y / y\n",
    "\n",
    "print(z)  # prints nan"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "807a2996-68dd-401a-856d-3a179676a303",
   "metadata": {},
   "source": [
    "Explanation:\r\n",
    "In this example, y is too large to be accurately represented as a float32. Therefore, it is stored as infinity (inf). When dividing x * y by y, we encounter a situation where infinity divided by infinity, which results in nan."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c13ca013-c6ef-419e-a0ab-7aff969d30d5",
   "metadata": {},
   "source": [
    "#### Example 3: Computing Smallest Positive Value and Maximum Value in float32 (using NumPy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "344f3f63-ad6b-468c-b1b1-61798a75df9d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1e-45\n",
      "3.4028235e+38\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# Compute the smallest positive value and maximum value in float32\n",
    "smallest_positive = np.nextafter(np.float32(0), np.float32(1))\n",
    "max_value = np.finfo(np.float32).max\n",
    "\n",
    "print(smallest_positive)  # prints 1.4013e-45\n",
    "print(max_value)  # prints 3.40282e+38"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "acde8d8d-667d-4096-8a3b-70b2b140462b",
   "metadata": {},
   "source": [
    "Explanation:\r\n",
    "In float32, the smallest positive value that can be represented is approximately 1.4013e-45, and the maximum value is approximately 3.40282e+38."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0e2f7e0-3e7e-4b87-aa90-047286fd9aa4",
   "metadata": {},
   "source": [
    "#### Example 4: Numerically Unstable Softmax Implementation (using PyTorch)\r\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "7a92f51e-0c6c-41d1-b08e-1cbefcfdb7f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nan  0.]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "def unstable_softmax(logits):\n",
    "    exp = torch.exp(logits)\n",
    "    return exp / torch.sum(exp)\n",
    "\n",
    "# Test the unstable softmax implementation\n",
    "print(unstable_softmax(torch.tensor([1000., 0.])).numpy())  # prints [ nan, 0.]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21335f98-6088-4206-b35b-d2e0b35aaf7f",
   "metadata": {},
   "source": [
    "Explanation:\r\n",
    "In this example, computing the exponential of very large logits results in values that are out of the range of float32, leading to numerical instability and producing nan."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00337a58-e359-4d31-9e03-79c7392907ca",
   "metadata": {},
   "source": [
    "#### Example 5: Stable Softmax Implementation (using PyTorch)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1b43c653-4918-49c3-ad4d-088c07f0406a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1. 0.]\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "def softmax(logits):\n",
    "    exp = torch.exp(logits - torch.max(logits))\n",
    "    return exp / torch.sum(exp)\n",
    "\n",
    "# Test the stable softmax implementation\n",
    "print(softmax(torch.tensor([1000., 0.])).numpy())  # prints [ 1., 0.]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2e41ac5-22f7-460d-bb74-ec717bcbcc71",
   "metadata": {},
   "source": [
    "Explanation:\r\n",
    "To ensure numerical stability, we subtract the maximum logit value from all logits before computing the softmax. This ensures that the range of the exponential function is limited to [-inf, 0], resulting in a stable computation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5cc343fc-8420-4dd7-a3ba-8ca2848f78cc",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.18"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}