examples/Calibration_for_CrossValidation_Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "# Perform Calibration Measurement for Cross Validated Models\n",
    "\n",
    "In this example, we demonstrate how to perform Calibration measurements (both 1-calibration and D-calibration) for cross validated models.\n",
    "\n",
    "One possible choice could be performing calibration evaluation on each fold and count how many times does the model calibrated.\n",
    "For example, say the D-calibration p-values for 5-fold cross validation are [0.01, 0.10, 0.98, 0.75, 0.25].\n",
    "We can use the threshold of 0.05 to conclude that the model is calibrated 4 times.\n",
    "\n",
    "On the other hand, if one wants to get an overall performance score on the whole dataset, it is not appropriate to average on the p-values over the folds (like we normally compute a mean and standard deviation for concordance or Brier scores).\n",
    "It is because p-values are not designed to be compared (it should only be used to accept or reject the hypothesis).\n",
    "Instead, we can combine the predicted survival curves from all cross-validation folds, and perform the calibration test on the total curves.\n",
    "This method is suggested in section 4.1 at [Haider's JMLR paper]((https://jmlr.org/papers/volume21/18-772/18-772.pdf)).\n",
    "\n",
    "This tutorial will implement the second method described above, which will only result a single p-value at the end."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "metadata": {},
   "cell_type": "code",
   "outputs": [],
   "execution_count": null,
   "source": "!pip install lifelines"
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "outputs": [],
   "source": [
    "from lifelines import CoxPHFitter\n",
    "from lifelines.datasets import load_rossi\n",
    "\n",
    "from tqdm import tqdm\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from Evaluator import LifelinesEvaluator\n",
    "from Evaluations.util import stratified_folds_survival"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    },
    "ExecuteTime": {
     "end_time": "2023-06-08T04:01:25.763724359Z",
     "start_time": "2023-06-08T04:01:24.457524411Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "outputs": [],
   "source": [
    "%matplotlib inline"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    },
    "ExecuteTime": {
     "end_time": "2023-06-08T04:41:46.873599338Z",
     "start_time": "2023-06-08T04:41:46.829814759Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Load the dataset."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "outputs": [],
   "source": [
    "np.random.seed(42)\n",
    "rossi = load_rossi()"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    },
    "ExecuteTime": {
     "end_time": "2023-06-08T04:41:48.541687042Z",
     "start_time": "2023-06-08T04:41:48.522508701Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Perform 5-fold cross validation on the whole dataset.\n",
    "We divide the folds using customized function so that each fold will have roughly the same distribution of times and censorships.\n",
    "\n",
    "We iteratively concatenate the predicted survival curves in each fold. And we have to do the same thing for event indicators and event times of testing set."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5/5 [00:00<00:00, 30.76it/s]\n"
     ]
    }
   ],
   "source": [
    "total_surv = pd.DataFrame()\n",
    "total_durations = np.empty([0, ])\n",
    "total_events = np.empty([0, ])\n",
    "\n",
    "for data_train, data_test in tqdm(stratified_folds_survival(rossi, rossi.week.values, rossi.arrest.values)):\n",
    "    durations_test = data_test['week'].values\n",
    "    events_test = data_test['arrest'].values.astype('bool')\n",
    "    # Fit the model\n",
    "    cph = CoxPHFitter()\n",
    "    cph.fit(data_train, duration_col='week', event_col='arrest')\n",
    "    survival_curves = cph.predict_survival_function(data_test, list(range(1, 53)))\n",
    "    # Combine the curves\n",
    "    total_surv = pd.concat([total_surv, survival_curves], axis=1)\n",
    "    total_durations = np.concatenate([total_durations, durations_test], axis=0)\n",
    "    total_events = np.concatenate([total_events, events_test], axis=0)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    },
    "ExecuteTime": {
     "end_time": "2023-06-08T04:41:50.719179078Z",
     "start_time": "2023-06-08T04:41:50.551789700Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Build the Evaluator.\n",
    "Since none of the calibration measurements need training information, we can omit those and only input survival curves and testing information."
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "outputs": [],
   "source": [
    "eval = LifelinesEvaluator(total_surv, total_durations, total_events)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    },
    "ExecuteTime": {
     "end_time": "2023-06-08T04:41:52.281825530Z",
     "start_time": "2023-06-08T04:41:52.275156338Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Run one calibration test at 50% time for the whole data set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 0 Axes>"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAGwCAYAAABVdURTAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAAA1yUlEQVR4nO3de1xUdeL/8fdwR1DwEiCIgrfMSlERxB6mJYaWtnZV1w2l1va7SV7Qvmmtt9RQSyXLS2pSbZrad2sr2yWN1C5iKEpqGpri4paArgaKCTXM7w9/zjaBNYMDA8fX8/E4jwfnOm9mtuXtOZ85x2SxWCwCAAAwCDdXBwAAAHAmyg0AADAUyg0AADAUyg0AADAUyg0AADAUyg0AADAUyg0AADAUD1cHqGuVlZX67rvv1LhxY5lMJlfHAQAAdrBYLDp37pxCQ0Pl5vbr52auuXLz3XffKTw83NUxAABADZw4cUKtWrX61W2uuXLTuHFjSZfenCZNmrg4DQAAsEdpaanCw8Otf8d/zTVXbi5fimrSpAnlBgCABsaeISUMKAYAAIZCuQEAAIZCuQEAAIZyzY25sZfZbNaPP/7o6hi4Cp6ennJ3d3d1DABAHaPc/ILFYlFhYaG+//57V0eBEwQGBiokJIR7GgHANYRy8wuXi01QUJAaNWrEH8UGymKx6MKFCyouLpYktWzZ0sWJAAB1hXLzM2az2Vpsmjdv7uo4uEq+vr6SpOLiYgUFBXGJCgCuEQwo/pnLY2waNWrk4iRwlsufJeOnAODaQbmpBpeijIPPEgCuPZQbAABgKJQbAABgKJQbe5lMdTvVkoiICKWlpdXa8eua0X4fAMDVo9wYyIkTJ/Twww8rNDRUXl5eatOmjcaPH6///Oc/ro4GAECdodwYxLFjxxQdHa0jR47ozTff1DfffKMVK1YoMzNTcXFxOnPmjEtymc1mVVZWuuS1AQDXJsqNQYwdO1ZeXl7avHmz+vbtq9atW2vQoEH66KOP9O233+rpp5+2bnvu3DmNGDFCfn5+CgsL09KlS63rLBaLZs6cqdatW8vb21uhoaEaN26cdX15ebkmT56ssLAw+fn5KTY2Vtu2bbOuf/XVVxUYGKj33ntPnTt3lre3t1avXi0fH58qd30eP368br/9duv8Z599pj59+sjX11fh4eEaN26cysrKrOuLi4s1ZMgQ+fr6KjIyUmvXrnXiOwgAMApu4mcAZ86c0Ycffqi5c+dab1x3WUhIiEaOHKkNGzZo2bJlkqTnnntOTz31lGbNmqUPP/xQ48ePV8eOHTVgwAD97W9/0+LFi7V+/XrdeOONKiws1Jdffmk9XnJysg4ePKj169crNDRU77zzjgYOHKj9+/erQ4cOkqQLFy5o/vz5Wr16tZo3b65WrVpp+vTp+tvf/qZHHnlE0qUzOhs2bNDcuXMlSUePHtXAgQM1Z84crVmzRqdOnVJycrKSk5OVnp4uSRo9erS+++47bd26VZ6enho3bpz1DsQAgKtjmuW88Z6WGRanHasmKDcGcOTIEVksFt1www3Vrr/hhht09uxZnTp1SpJ0yy23aMqUKZKkjh076vPPP9fixYs1YMAAFRQUKCQkRPHx8fL09FTr1q0VExMjSSooKFB6eroKCgoUGhoqSZo8ebIyMjKUnp6uZ599VtKlG+YtW7ZMXbt2tWYYPny41q1bZy03mZmZ+v7773XfffdJklJTUzVy5EhNmDBBktShQwctWbJEffv21fLly1VQUKB//vOfys7OVs+ePSVJr7zyyhV/ZwDAtYvLUgZisdjXlOPi4qrMHzp0SJL0wAMP6IcfflDbtm01ZswYvfPOO/rpp58kSfv375fZbFbHjh3l7+9vnbZv366jR49aj+fl5aUuXbrYvMbIkSO1bds2fffdd5KktWvX6q677lJgYKAk6csvv9Srr75qc9yEhARVVlYqPz9fhw4dkoeHh3r06GE9ZqdOnaz7AwBwGWduDKB9+/YymUw6dOiQ7rnnnirrDx06pKZNm+q66677zWOFh4crLy9PH330kbZs2aLHHntMzz33nLZv367z58/L3d1dOTk5VZ7T5O/vb/3Z19e3yp2Be/bsqXbt2mn9+vX685//rHfeeUevvvqqdf358+f1pz/9yWZ8z2WtW7fW4cOHfzM7AAAS5cYQmjdvrgEDBmjZsmWaOHGizbibwsJCrV27VomJidbCsXPnTpv9d+7caXN5x9fXV0OGDNGQIUM0duxYderUSfv371e3bt1kNptVXFysPn36OJxz5MiRWrt2rVq1aiU3Nzfddddd1nXdu3fXwYMH1b59+2r37dSpk3766Sfl5ORYL0vl5eVVGaQMAACXpQzipZdeUnl5uRISEvTJJ5/oxIkTysjI0IABAxQWFmYduCtJn3/+uRYsWKDDhw9r6dKleuuttzR+/HhJl77t9Morr+jAgQM6duyY3njjDfn6+qpNmzbq2LGjRo4cqcTERL399tvKz89Xdna2UlNT9cEHH/xmxpEjR2rPnj2aO3eu7r//fnl7e1vXPfnkk9qxY4eSk5OVm5urI0eO6N1331VycrIk6frrr9fAgQP1pz/9SV988YVycnL0xz/+scoAagAAKDf2sljqdnJQhw4dtHv3brVt21YPPvig2rVrp0cffVS33XabsrKy1KxZM+u2kyZN0u7du9WtWzfNmTNHixYtUkJCgiQpMDBQq1at0i233KIuXbroo48+0vvvv6/mzZtLktLT05WYmKhJkybp+uuv19ChQ7Vr1y61bt36NzO2b99eMTEx2rdvn0aOHGmzrkuXLtq+fbsOHz6sPn36qFu3bpo+fbp14PLl1w4NDVXfvn1177336tFHH1VQUJDD7xUAwNhMFntHoRpEaWmpAgICVFJSoiZNmtisu3jxovLz8xUZGSkfHx8XJYQz8ZkCgH3q+1fBf+3v9y9x5gYAABgK5QYAABgK5QYAABgK5QYAABgK5QYAABgK5QYAABgK5QYAABgK5QYAABgK5QYNUr9+/TRhwgRXxwAA1EP14sGZS5cu1XPPPafCwkJ17dpVL774omJiYqrd9tVXX1VSUpLNMm9vb128eLFWMzrzzo32cPTujqNHj9Zrr71WZXlCQoIyMjKcFeuq9OvXT1FRUUpLS3N1FACAgbm83GzYsEEpKSlasWKFYmNjlZaWpoSEBOXl5V3xuUFNmjRRXl6edf7y066vdQMHDlR6errNsp8/nBIAgGuByy9LLVq0SGPGjFFSUpI6d+6sFStWqFGjRlqzZs0V9zGZTAoJCbFOwcHBdZi4/vL29rZ5X0JCQtS0aVNt27ZNXl5e+vTTT63bLliwQEFBQSoqKpJ06axKcnKykpOTFRAQoBYtWmjatGn6+aPHysvLNXnyZIWFhcnPz0+xsbHatm2bTYbPP/9c/fr1U6NGjdS0aVMlJCTo7NmzGj16tLZv364XXnhBJpNJJpNJx48flyQdOHBAgwYNkr+/v4KDg/XQQw/p9OnT1mOWlZUpMTFR/v7+atmypRYuXFh7byIAoMFzabmpqKhQTk6O4uPjrcvc3NwUHx+vrKysK+53/vx5tWnTRuHh4frd736nr7766orblpeXq7S01Ga61lwen/LQQw+ppKREe/fu1bRp07R69WqbYvjaa6/Jw8ND2dnZeuGFF7Ro0SKtXr3auj45OVlZWVlav3699u3bpwceeEADBw7UkSNHJEm5ubnq37+/OnfurKysLH322WcaMmSIzGazXnjhBcXFxWnMmDE6efKkTp48qfDwcH3//fe6/fbb1a1bN+3evVsZGRkqKirSgw8+aH3dJ554Qtu3b9e7776rzZs3a9u2bdqzZ0/dvYEAgAbFpZelTp8+LbPZXOXMS3BwsL7++utq97n++uu1Zs0adenSRSUlJXr++efVu3dvffXVV2rVqlWV7VNTUzVr1qxayV/fbNq0Sf7+/jbLnnrqKT311FOaM2eOtmzZokcffVQHDhzQqFGjdPfdd9tsGx4ersWLF8tkMun666/X/v37tXjxYo0ZM0YFBQVKT09XQUGBQkNDJUmTJ09WRkaG0tPT9eyzz2rBggWKjo7WsmXLrMe88cYbrT97eXmpUaNGCgkJsS576aWX1K1bNz377LPWZWvWrFF4eLgOHz6s0NBQvfLKK3rjjTfUv39/SZdKWHWfNQAAUj0Yc+OouLg4xcXFWed79+6tG264QS+//LJmz55dZfupU6cqJSXFOl9aWqrw8PA6yVrXbrvtNi1fvtxmWbNmzSRdKhZr165Vly5d1KZNGy1evLjK/r169bIZvxQXF6eFCxfKbDZr//79MpvN6tixo80+5eXlat68uaRLZ24eeOABhzJ/+eWX2rp1a5VSJklHjx7VDz/8oIqKCsXGxtr8Ttdff71DrwMAuHa4tNy0aNFC7u7u1nEflxUVFdn86/7XeHp6qlu3bvrmm2+qXe/t7X3NDKr18/NT+/btr7h+x44dkqQzZ87ozJkz8vPzs/vY58+fl7u7u3JycuTu7m6z7nIx8fX1dTjz+fPnNWTIEM2fP7/KupYtW17xcwUA4EpcOubGy8tLPXr0UGZmpnVZZWWlMjMzbc7O/JrLZxVatmxZWzEN4ejRo5o4caJWrVql2NhYjRo1SpWVlTbbfPHFFzbzO3fuVIcOHeTu7q5u3brJbDaruLhY7du3t5kuF9EuXbrYfJa/5OXlJbPZbLOse/fu+uqrrxQREVHluH5+fmrXrp08PT1tsp09e1aHDx++2rcEAGBQLv+2VEpKilatWqXXXntNhw4d0p///GeVlZVZ72WTmJioqVOnWrd/5plntHnzZh07dkx79uzRH/7wB/3rX//SH//4R1f9CvVGeXm5CgsLbabL45r+8Ic/KCEhQUlJSUpPT9e+ffuqfOuooKBAKSkpysvL05tvvqkXX3xR48ePlyR17NhRI0eOVGJiot5++23l5+crOztbqamp+uCDDyRdugS4a9cuPfbYY9q3b5++/vprLV++3PrNp4iICH3xxRc6fvy4Tp8+rcrKSo0dO1ZnzpzRiBEjtGvXLh09elQffvihkpKSZDab5e/vr0ceeURPPPGEPv74Yx04cECjR4+Wm5vL/6cLAKinXD7mZtiwYTp16pSmT5+uwsJCRUVFKSMjwzrIuKCgwOYP2dmzZzVmzBgVFhaqadOm6tGjh3bs2KHOnTu76leoNzIyMqqcwbr++uv1+9//Xv/617+0adMmSZcu96xcuVIjRozQHXfcoa5du0q6VCR/+OEHxcTEyN3dXePHj9ejjz5qPVZ6errmzJmjSZMm6dtvv1WLFi3Uq1cvDR48WNKlArR582Y99dRTiomJka+vr2JjYzVixAhJlwYgjxo1Sp07d9YPP/yg/Px8RURE6PPPP9eTTz6pO+64Q+Xl5WrTpo0GDhxo/dyfe+456+Wrxo0ba9KkSSopKan19xMA0DCZLD+/kck1oLS0VAEBASopKVGTJk1s1l28eFH5+fmKjIyUj4+PixK6hlHvHnwtf6YA4Ahn3onf0bvs2+PX/n7/Euf2AQCAoVBuAACAobh8zA3qh18+RgEAgIaKMzcAAMBQKDfVuMbGWBsanyUAXHsoNz/j6ekpSbpw4YKLk8BZLn+Wlz9bAIDxMebmZ9zd3RUYGKji4mJJUqNGjWyetYSGw2Kx6MKFCyouLlZgYGCVR0YAAIyLcvMLlx8lcLngoGELDAy0+zllAABjoNz8gslkUsuWLRUUFKQff/zR1XFwFTw9PTljA8D4nHWFYaZzDlMfUG6uwN3dnT+MAAA0QAwoBgAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhkK5AQAAhlIvys3SpUsVEREhHx8fxcbGKjs726791q9fL5PJpKFDh9ZuQAAA0GC4vNxs2LBBKSkpmjFjhvbs2aOuXbsqISFBxcXFv7rf8ePHNXnyZPXp06eOkgIAgIbA5eVm0aJFGjNmjJKSktS5c2etWLFCjRo10po1a664j9ls1siRIzVr1iy1bdu2DtMCAID6zqXlpqKiQjk5OYqPj7cuc3NzU3x8vLKysq643zPPPKOgoCA98sgjv/ka5eXlKi0ttZkAAIBxubTcnD59WmazWcHBwTbLg4ODVVhYWO0+n332mV555RWtWrXKrtdITU1VQECAdQoPD7/q3AAAoP5y+WUpR5w7d04PPfSQVq1apRYtWti1z9SpU1VSUmKdTpw4UcspAQCAK3m48sVbtGghd3d3FRUV2SwvKipSSEhIle2PHj2q48ePa8iQIdZllZWVkiQPDw/l5eWpXbt2Nvt4e3vL29u7FtIDAID6yKVnbry8vNSjRw9lZmZal1VWViozM1NxcXFVtu/UqZP279+v3Nxc63T33XfrtttuU25uLpecAACAa8/cSFJKSopGjRql6OhoxcTEKC0tTWVlZUpKSpIkJSYmKiwsTKmpqfLx8dFNN91ks39gYKAkVVkOAACuTS4vN8OGDdOpU6c0ffp0FRYWKioqShkZGdZBxgUFBXJza1BDgwAAgAuZLBaLxdUh6lJpaakCAgJUUlKiJk2auDoOAABXx2RyzmFmOuUwkiTLDOdXC0f+fnNKBAAAGArlBgAAGArlBgAAGArlBgAAGArlBgAAGArlBgAAGIrL73MDAICRmGY556vZUu18pfpawJkbAABgKA6Xm759++r111/XDz/8UBt5AAAArorD5aZbt26aPHmyQkJCNGbMGO3cubM2cgEAANSIw+UmLS1N3333ndLT01VcXKxbb71VnTt31vPPP6+ioqLayAgAAGC3Go258fDw0L333qt3331X//73v/X73/9e06ZNU3h4uIYOHaqPP/7Y2TkBAADsclUDirOzszVjxgwtXLhQQUFBmjp1qlq0aKHBgwdr8uTJzsoIAABgN4e/Cl5cXKy//vWvSk9P15EjRzRkyBC9+eabSkhIkOn/P5l09OjRGjhwoJ5//nmnBwYAAPg1DpebVq1aqV27dnr44Yc1evRoXXfddVW26dKli3r27OmUgAAAAI5wuNxkZmaqT58+v7pNkyZNtHXr1hqHAgAAqCmHx9zMmDFD33//fZXlpaWluv32252RCQAAoMYcLjfbt29XRUVFleUXL17Up59+6pRQAAAANWX3Zal9+/ZJkiwWiw4ePKjCwkLrOrPZrIyMDIWFhTk/IQAAgAPsLjdRUVEymUwymUzVXn7y9fXViy++6NRwAAAAjrK73OTn58tisaht27bKzs62+ZaUl5eXgoKC5O7uXishAQAA7GV3uWnTpo0kqbKystbCAAAAXC27ys17772nQYMGydPTU++9996vbnv33Xc7JRgAAEBN2FVuhg4dqsLCQgUFBWno0KFX3M5kMslsNjsrGwAAgMPsKjc/vxTFZSkAAFCfXdWDMwEAAOobu87cLFmyxO4Djhs3rsZhAAAArpZd5Wbx4sV2HcxkMlFuAACAS9lVbvLz82s7BwAAgFMw5gYAABiKXWduUlJSNHv2bPn5+SklJeVXt120aJFTggEAANSEXeVm7969+vHHH60/X4nJZHJOKgAAgBqyq9xs3bq12p8BAADqm6sac3PixAmdOHHCWVkAAACumsPl5qefftK0adMUEBCgiIgIRUREKCAgQH/5y1+sl64AAABcxe6ngl/2+OOP6+2339aCBQsUFxcnScrKytLMmTP1n//8R8uXL3d6SAAAAHs5XG7WrVun9evXa9CgQdZlXbp0UXh4uEaMGEG5AQAALuXwZSlvb29FRERUWR4ZGSkvLy9nZAIAAKgxh8tNcnKyZs+erfLycuuy8vJyzZ07V8nJyU4NBwAA4Ci7Lkvde++9NvMfffSRWrVqpa5du0qSvvzyS1VUVKh///7OTwgAAOAAu8pNQECAzfx9991nMx8eHu68RAAAAFfBrnKTnp5e2zkAALBhmuW8u95bZlicdizUfzw4EwAAGIrDXwWXpP/7v//Txo0bVVBQoIqKCpt1e/bscUowAACAmnD4zM2SJUuUlJSk4OBg7d27VzExMWrevLmOHTtmc+8bAAAAV3C43CxbtkwrV67Uiy++KC8vL/3v//6vtmzZonHjxqmkpKQ2MgIAANjN4XJTUFCg3r17S5J8fX117tw5SdJDDz2kN99807npAAAAHORwuQkJCdGZM2ckSa1bt9bOnTslSfn5+bJYGI0OAGigTCbnTHA5h8vN7bffrvfee0+SlJSUpIkTJ2rAgAEaNmyY7rnnHqcHBAAAcITD35ZauXKlKisrJUljx45V8+bNtWPHDt19993605/+5PSAAAAAjnC43Li5ucnN7b8nfIYPH67hw4c7NRQAAEBN1eg+N2fPntUrr7yiQ4cOSZI6d+6spKQkNWvWzKnhAAAAHOXwmJtPPvlEkZGRWrJkic6ePauzZ89qyZIlioyM1CeffFIbGQEAAOzm8JmbsWPH6sEHH9Ty5cvl7u4uSTKbzXrsscc0duxY7d+/3+khAQAA7OXwmZtvvvlGkyZNshYbSXJ3d1dKSoq++eYbp4YDAABwlMPlpnv37taxNj936NAhde3atUYhli5dqoiICPn4+Cg2NlbZ2dlX3Pbtt99WdHS0AgMD5efnp6ioKP31r3+t0esCAADjseuy1L59+6w/jxs3TuPHj9c333yjXr16SZJ27typpUuXat68eQ4H2LBhg1JSUrRixQrFxsYqLS1NCQkJysvLU1BQUJXtmzVrpqefflqdOnWSl5eXNm3apKSkJAUFBSkhIcHh1wcAAMZisthxW2E3NzeZTKbfvAOxyWSS2Wx2KEBsbKx69uypl156SZJUWVmp8PBwPf7445oyZYpdx+jevbvuuusuzZ49+ze3LS0tVUBAgEpKStSkSROHsgIA6o5plvPu9muZYccd9J10d2HTTKccRpLBczvIkb/fdp25yc/Pd0qwX6qoqFBOTo6mTp1qXebm5qb4+HhlZWX95v4Wi0Uff/yx8vLyNH/+/Gq3KS8vV3l5uXW+tLT06oMDAK7MWY8gmOmcw+DaY1e5adOmTa28+OnTp2U2mxUcHGyzPDg4WF9//fUV9yspKVFYWJjKy8vl7u6uZcuWacCAAdVum5qaqlmzZjk1NwAAqL9qdBO/o0ePKi0tzeYmfuPHj1e7du2cGu5KGjdurNzcXJ0/f16ZmZlKSUlR27Zt1a9fvyrbTp06VSkpKdb50tJShYeH10lOAABQ9xwuNx9++KHuvvtuRUVF6ZZbbpEkff7557rxxhv1/vvvX/EMSnVatGghd3d3FRUV2SwvKipSSEjIFfdzc3NT+/btJUlRUVE6dOiQUlNTqy033t7e8vb2tjsTAABo2BwuN1OmTNHEiROrfDNqypQpevLJJx0qN15eXurRo4cyMzM1dOhQSZcGFGdmZio5Odnu41RWVtqMqwEAANcuh8vNoUOHtHHjxirLH374YaWlpTkcICUlRaNGjVJ0dLRiYmKUlpamsrIyJSUlSZISExMVFham1NRUSZfG0ERHR6tdu3YqLy/XP/7xD/31r3/V8uXLHX5tALgW1Pm3jgAXc7jcXHfddcrNzVWHDh1slufm5lZ7X5rfMmzYMJ06dUrTp09XYWGhoqKilJGRYR1kXFBQYPMU8rKyMj322GP697//LV9fX3Xq1ElvvPGGhg0b5vBrAwAA43G43IwZM0aPPvqojh07pt69e0u6NOZm/vz5NgN3HZGcnHzFy1Dbtm2zmZ8zZ47mzJlTo9cBAADG53C5mTZtmho3bqyFCxda708TGhqqmTNnaty4cU4PCAAA4AiHys1PP/2kdevW6fe//70mTpyoc+fOSbr01WwAAID6wKEHZ3p4eOh//ud/dPHiRUmXSg3FBgAA1CcOPxU8JiZGe/furY0sAAAAV83hMTePPfaYJk2apH//+9/q0aOH/Pz8bNZ36dLFaeEAAAAc5XC5GT58uCTZDB6+/MTwmjwVHAAAwJkcLje19YRwAAAAZ3Co3JSWlurw4cOqqKhQTEyMrrvuutrKBQDOY3LSHXot3J0XaAjsLje5ubm68847VVRUJIvFosaNG2vjxo1KSEiozXwAAAAOsfvbUk8++aQiIyP12WefKScnR/3793fo4ZYAAAB1we4zNzk5Odq8ebO6d+8uSVqzZo2aNWum0tJSNWnSpNYCAgAAOMLuMzdnzpxRq1atrPOBgYHy8/PTf/7zn1oJBgAAUBMODSg+ePCgCgsLrfMWi0WHDh2yPoZB4j43AADAtRwqN/3795flF98WGDx4MPe5AQAA9Ybd5Yb72wAAgIbA7nLTpk2b2swBAPWeaZaT7pcjyTKDe+YAtcXhB2cCAADUZ5QbAABgKJQbAABgKJQbAABgKJQbAABgKHZ9W6pbt24y2flU3T179lxVIAAAgKthV7kZOnRoLccA0CDY+Y+c32Tha9AAao9d5WbGjBm1nQMAAMApGHMDAAAMxaFnS0mS2WzW4sWLtXHjRhUUFKiiosJm/ZkzZ5wWDgAAwFEOn7mZNWuWFi1apGHDhqmkpEQpKSm699575ebmppkzZ9ZCRAAAAPs5XG7Wrl2rVatWadKkSfLw8NCIESO0evVqTZ8+XTt37qyNjAAAAHZzuNwUFhbq5ptvliT5+/urpKREkjR48GB98MEHzk0HAADgIIfLTatWrXTy5ElJUrt27bR582ZJ0q5du+Tt7e3cdAAAAA5yuNzcc889yszMlCQ9/vjjmjZtmjp06KDExEQ9/PDDTg8IAADgCIe/LTVv3jzrz8OGDVObNm20Y8cOdejQQUOGDHFqOAAAAEc5XG4uXrwoHx8f63yvXr3Uq1cvp4YCAACoKYcvSwUFBWnUqFHasmWLKisrayMTAABAjTlcbl577TVduHBBv/vd7xQWFqYJEyZo9+7dtZENAADAYTUaUPzWW2+pqKhIzz77rA4ePKhevXqpY8eOeuaZZ2ojIwAAgN0cHnNzWePGjZWUlKSkpCQdPHhQI0eO1KxZszR9+nRn5gNgQKZZTnq6uCTLDJ4wDsBWjR+cefHiRW3cuFFDhw5V9+7ddebMGT3xxBPOzAYAAOAwh8/cfPjhh1q3bp3+/ve/y8PDQ/fff782b96sW2+9tTbyoa6YnPQvaQv/igacxln/Xc50zmGAhsLhcnPPPfdo8ODBev3113XnnXfK09OzNnIBAADUiMPlpqioSI0bN66NLAAAAFfNrnJTWlqqJk2aSJIsFotKS0uvuO3l7QAAAFzBrnLTtGlTnTx5UkFBQQoMDJSpmuvAFotFJpNJZrPZ6SEBAADsZVe5+fjjj9WsWTPrz9WVGwAAgPrArnLTt29f68/9+vWrrSwAAABXzeH73HTo0EEzZ87UkSNHaiMPAADAVXG43Dz22GP64IMP1KlTJ/Xs2VMvvPCCCgsLayMbAACAwxwuNxMnTtSuXbt06NAh3XnnnVq6dKnCw8N1xx136PXXX6+NjAAAAHar8eMXOnbsqFmzZunw4cP69NNPderUKSUlJTkzGwAAgMNq/OBMScrOzta6deu0YcMGlZaW6oEHHnBWLgAAgBpxuNwcPnxYa9eu1Ztvvqn8/Hzdfvvtmj9/vu699175+/vXRkYAAAC7OVxuLg8kHjt2rIYPH67g4ODayAUAAFAjDpUbs9msl19+Wffff7+aNm1aW5kAAABqzKEBxe7u7nr88cf1/fff11IcwEEmk3MmcgOAYTh8Weqmm27SsWPHFBkZWRt5ADjANMt5Bccyw+K0YwGAKzn8VfA5c+Zo8uTJ2rRpk06ePKnS0lKbCQAAwJUcLjd33nmnvvzyS919991q1aqVmjZtqqZNmyowMLDG43CWLl2qiIgI+fj4KDY2VtnZ2VfcdtWqVerTp4/1dePj4391ewAAcG1x+LLU1q1bnRpgw4YNSklJ0YoVKxQbG6u0tDQlJCQoLy9PQUFBVbbftm2bRowYod69e8vHx0fz58/XHXfcoa+++kphYWFOzQYAABoeh8vNz58Q7gyLFi3SmDFjrHc3XrFihT744AOtWbNGU6ZMqbL92rVrbeZXr16tv/3tb8rMzFRiYmKV7cvLy1VeXm6d59IZAADG5nC5+eSTT351/a233mr3sSoqKpSTk6OpU6dal7m5uSk+Pl5ZWVl2HePChQv68ccf1axZs2rXp6amatasWXZnAgAADZvD5aZfv35Vlpl+9pVUs9ls97FOnz4ts9lc5UaAwcHB+vrrr+06xpNPPqnQ0FDFx8dXu37q1KlKSUmxzpeWlio8PNzujAAAoGFxuNycPXvWZv7HH3/U3r17NW3aNM2dO9dpwewxb948rV+/Xtu2bZOPj0+123h7e8vb27tOcwEAANdxuNwEBARUWTZgwAB5eXkpJSVFOTk5dh+rRYsWcnd3V1FRkc3yoqIihYSE/Oq+zz//vObNm6ePPvpIXbp0sfs1AQCAsTn8VfArCQ4OVl5enkP7eHl5qUePHsrMzLQuq6ysVGZmpuLi4q6434IFCzR79mxlZGQoOjq6xpkBAIDxOHzmZt++fTbzFotFJ0+e1Lx58xQVFeVwgJSUFI0aNUrR0dGKiYlRWlqaysrKrN+eSkxMVFhYmFJTUyVJ8+fP1/Tp07Vu3TpFRESosLBQkuTv789TyQEAgOPlJioqSiaTSRaL7a3ae/XqpTVr1jgcYNiwYTp16pSmT5+uwsJCRUVFKSMjwzrIuKCgQG5u/z3BtHz5clVUVOj++++3Oc6MGTM0c+ZMh18fAAAYi8PlJj8/32bezc1N11133RUH9NojOTlZycnJ1a7btm2bzfzx48dr/DoAAMD4HC43bdq0qY0cAAAATmH3gOKsrCxt2rTJZtnrr7+uyMhIBQUF6dFHH7W5EzAAAIAr2F1unnnmGX311VfW+f379+uRRx5RfHy8pkyZovfff9866BcAAMBV7C43ubm56t+/v3V+/fr1io2N1apVq5SSkqIlS5Zo48aNtRISAADAXnaXm7Nnz9o8JmH79u0aNGiQdb5nz546ceKEc9MBAAA4yO5yExwcbP2mVEVFhfbs2aNevXpZ1587d06enp7OTwgAAOAAu8vNnXfeqSlTpujTTz/V1KlT1ahRI/Xp08e6ft++fWrXrl2thAQAALCX3V8Fnz17tu6991717dtX/v7+eu211+Tl5WVdv2bNGt1xxx21EhIAAMBedpebFi1a6JNPPlFJSYn8/f3l7u5us/6tt97i8QcAAMDlnPJUcElq1qzZVYcBXMU0y+S0Y1lmWH57IwBArXHaU8EBAADqA8oNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFMoNAAAwFA9XB4CxmGaZnHYsywyL044FALh2cOYGAAAYCuUGAAAYCuUGAAAYCuUGAAAYCuUGAAAYCuUGAAAYCuUGAAAYCuUGAAAYCuUGAAAYCuUGAAAYCo9fcDaTkx4/YOHRAwAA1ARnbgAAgKFQbgAAgKFQbgAAgKFQbgAAgKFQbgAAgKFQbgAAgKG4vNwsXbpUERER8vHxUWxsrLKzs6+47VdffaX77rtPERERMplMSktLq7ugAACgQXBpudmwYYNSUlI0Y8YM7dmzR127dlVCQoKKi4ur3f7ChQtq27at5s2bp5CQkDpOCwAAGgKXlptFixZpzJgxSkpKUufOnbVixQo1atRIa9asqXb7nj176rnnntPw4cPl7e1t12uUl5ertLTUZgIAAMblsnJTUVGhnJwcxcfH/zeMm5vi4+OVlZXltNdJTU1VQECAdQoPD3fasQEAQP3jsnJz+vRpmc1mBQcH2ywPDg5WYWGh015n6tSpKikpsU4nTpxw2rEBAED9Y/hnS3l7e9t9CQsAADR8Ljtz06JFC7m7u6uoqMhmeVFREYOFAQBAjbms3Hh5ealHjx7KzMy0LqusrFRmZqbi4uJcFQsAADRwLr0slZKSolGjRik6OloxMTFKS0tTWVmZkpKSJEmJiYkKCwtTamqqpEuDkA8ePGj9+dtvv1Vubq78/f3Vvn17l/0eAACg/nBpuRk2bJhOnTql6dOnq7CwUFFRUcrIyLAOMi4oKJCb239PLn333Xfq1q2bdf7555/X888/r759+2rbtm11HR8AANRDLh9QnJycrOTk5GrX/bKwREREyGKx1EEqAADQULn88QsAAADORLkBAACGQrkBAACGQrkBAACGQrkBAACGQrkBAACGQrkBAACG4vL73KB6plkmpx3LMoN7AwEArh2cuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZCuQEAAIZSL8rN0qVLFRERIR8fH8XGxio7O/tXt3/rrbfUqVMn+fj46Oabb9Y//vGPOkoKAADqO5eXmw0bNiglJUUzZszQnj171LVrVyUkJKi4uLja7Xfs2KERI0bokUce0d69ezV06FANHTpUBw4cqOPkAACgPnJ5uVm0aJHGjBmjpKQkde7cWStWrFCjRo20Zs2aard/4YUXNHDgQD3xxBO64YYbNHv2bHXv3l0vvfRSHScHAAD1kYcrX7yiokI5OTmaOnWqdZmbm5vi4+OVlZVV7T5ZWVlKSUmxWZaQkKC///3v1W5fXl6u8vJy63xJSYkkqbS09CrT17KLzjtUnf6u5Ca3PchNbnuQm9zVHNNisfz2xhYX+vbbby2SLDt27LBZ/sQTT1hiYmKq3cfT09Oybt06m2VLly61BAUFVbv9jBkzLJKYmJiYmJiYDDCdOHHiN/uFS8/c1IWpU6fanOmprKzUmTNn1Lx5c5lMJhcmM57S0lKFh4frxIkTatKkiavjGB7vd93i/a5bvN91qyG83xaLRefOnVNoaOhvbuvSctOiRQu5u7urqKjIZnlRUZFCQkKq3SckJMSh7b29veXt7W2zLDAwsOah8ZuaNGlSb//jMCLe77rF+123eL/rVn1/vwMCAuzazqUDir28vNSjRw9lZmZal1VWViozM1NxcXHV7hMXF2ezvSRt2bLlitsDAIBri8svS6WkpGjUqFGKjo5WTEyM0tLSVFZWpqSkJElSYmKiwsLClJqaKkkaP368+vbtq4ULF+quu+7S+vXrtXv3bq1cudKVvwYAAKgnXF5uhg0bplOnTmn69OkqLCxUVFSUMjIyFBwcLEkqKCiQm9t/TzD17t1b69at01/+8hc99dRT6tChg/7+97/rpptuctWvgP/P29tbM2bMqHIZELWD97tu8X7XLd7vumW099tksdjznSoAAICGweU38QMAAHAmyg0AADAUyg0AADAUyg0AADAUyg2uWmpqqnr27KnGjRsrKChIQ4cOVV5enqtjXTPmzZsnk8mkCRMmuDqKYX377bf6wx/+oObNm8vX11c333yzdu/e7epYhmQ2mzVt2jRFRkbK19dX7dq10+zZs+17nhB+0yeffKIhQ4YoNDRUJpOpynMZLRaLpk+frpYtW8rX11fx8fE6cuSIa8JeBcoNrtr27ds1duxY7dy5U1u2bNGPP/6oO+64Q2VlZa6OZni7du3Syy+/rC5durg6imGdPXtWt9xyizw9PfXPf/5TBw8e1MKFC9W0aVNXRzOk+fPna/ny5XrppZd06NAhzZ8/XwsWLNCLL77o6miGUFZWpq5du2rp0qXVrl+wYIGWLFmiFStW6IsvvpCfn58SEhJ08aITn6pZB/gqOJzu1KlTCgoK0vbt23Xrrbe6Oo5hnT9/Xt27d9eyZcs0Z84cRUVFKS0tzdWxDGfKlCn6/PPP9emnn7o6yjVh8ODBCg4O1iuvvGJddt9998nX11dvvPGGC5MZj8lk0jvvvKOhQ4dKunTWJjQ0VJMmTdLkyZMlSSUlJQoODtarr76q4cOHuzCtYzhzA6crKSmRJDVr1szFSYxt7NixuuuuuxQfH+/qKIb23nvvKTo6Wg888ICCgoLUrVs3rVq1ytWxDKt3797KzMzU4cOHJUlffvmlPvvsMw0aNMjFyYwvPz9fhYWFNv+fEhAQoNjYWGVlZbkwmeNcfodiGEtlZaUmTJigW265hbtG16L169drz5492rVrl6ujGN6xY8e0fPlypaSk6KmnntKuXbs0btw4eXl5adSoUa6OZzhTpkxRaWmpOnXqJHd3d5nNZs2dO1cjR450dTTDKywslCTrEwIuCw4Otq5rKCg3cKqxY8fqwIED+uyzz1wdxbBOnDih8ePHa8uWLfLx8XF1HMOrrKxUdHS0nn32WUlSt27ddODAAa1YsYJyUws2btyotWvXat26dbrxxhuVm5urCRMmKDQ0lPcbduOyFJwmOTlZmzZt0tatW9WqVStXxzGsnJwcFRcXq3v37vLw8JCHh4e2b9+uJUuWyMPDQ2az2dURDaVly5bq3LmzzbIbbrhBBQUFLkpkbE888YSmTJmi4cOH6+abb9ZDDz2kiRMnWh+ejNoTEhIiSSoqKrJZXlRUZF3XUFBucNUsFouSk5P1zjvv6OOPP1ZkZKSrIxla//79tX//fuXm5lqn6OhojRw5Urm5uXJ3d3d1REO55ZZbqtza4PDhw2rTpo2LEhnbhQsXbB6WLEnu7u6qrKx0UaJrR2RkpEJCQpSZmWldVlpaqi+++EJxcXEuTOY4Lkvhqo0dO1br1q3Tu+++q8aNG1uvzQYEBMjX19fF6YyncePGVcYz+fn5qXnz5oxzqgUTJ05U79699eyzz+rBBx9Udna2Vq5cqZUrV7o6miENGTJEc+fOVevWrXXjjTdq7969WrRokR5++GFXRzOE8+fP65tvvrHO5+fnKzc3V82aNVPr1q01YcIEzZkzRx06dFBkZKSmTZum0NBQ6zeqGgwLcJUkVTulp6e7Oto1o2/fvpbx48e7OoZhvf/++5abbrrJ4u3tbenUqZNl5cqVro5kWKWlpZbx48dbWrdubfHx8bG0bdvW8vTTT1vKy8tdHc0Qtm7dWu3/X48aNcpisVgslZWVlmnTplmCg4Mt3t7elv79+1vy8vJcG7oGuM8NAAAwFMbcAAAAQ6HcAAAAQ6HcAAAAQ6HcAAAAQ6HcAAAAQ6HcAAAAQ6HcAAAAQ6HcAAAAQ6HcADCM48ePy2QyKTc319VRALgQ5QZAgzF69GiZTCbr1Lx5cw0cOFD79u2TJIWHh+vkyZM8Ywu4xlFuADQoAwcO1MmTJ3Xy5EllZmbKw8NDgwcPlnTp6dEhISHy8OCZwMC1jHIDoEHx9vZWSEiIQkJCFBUVpSlTpujEiRM6depUlctS27Ztk8lkUmZmpqKjo9WoUSP17t1beXl5rv0lANQqyg2ABuv8+fN644031L59ezVv3vyK2z399NNauHChdu/eLQ8PDz388MN1mBJAXePcLYAGZdOmTfL395cklZWVqWXLltq0aZPc3K78b7W5c+eqb9++kqQpU6borrvu0sWLF+Xj41MnmQHULc7cAGhQbrvtNuXm5io3N1fZ2dlKSEjQoEGD9K9//euK+3Tp0sX6c8uWLSVJxcXFtZ4VgGtQbgA0KH5+fmrfvr3at2+vnj17avXq1SorK9OqVauuuI+np6f1Z5PJJEmqrKys9awAXINyA6BBM5lMcnNz0w8//ODqKADqCcbcAGhQysvLVVhYKEk6e/asXnrpJZ0/f15DhgxxcTIA9QXlBkCDkpGRYR0307hxY3Xq1ElvvfWW+vXrp+PHj7s2HIB6wWSxWCyuDgEAAOAsjLkBAACGQrkBAACGQrkBAACGQrkBAACGQrkBAACGQrkBAACGQrkBAACGQrkBAACGQrkBAACGQrkBAACGQrkBAACG8v8APkzGD6HyYNoAAAAASUVORK5CYII="
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross validated model is one-calibrated at time 52.0.\n"
     ]
    }
   ],
   "source": [
    "one_cal_times = np.quantile(rossi['week'], 0.5)\n",
    "\n",
    "p_value, obs_probs, exp_probs = eval.one_calibration(target_time=one_cal_times, method=\"DN\")\n",
    "# Returned probability are ranked from high to low. We want to draw the calibration bar plot from low to high.\n",
    "bins = [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]\n",
    "\n",
    "plt.clf()\n",
    "fig, ax = plt.subplots()\n",
    "bar1 = ax.bar([x - 0.2 for x in bins], obs_probs, width=0.4, color='r', align='center')\n",
    "bar2 = ax.bar([x + 0.2 for x in bins], exp_probs, width=0.4, color='g', align='center')\n",
    "\n",
    "ax.legend((bar1[0], bar2[0]), ('Observed', 'Expected'))\n",
    "plt.ylabel(\"Survival Probability\")\n",
    "plt.xlabel(\"Bin\")\n",
    "plt.show()\n",
    "if p_value >= 0.05:\n",
    "    print(\"The cross validated model is one-calibrated at time {}.\".format(one_cal_times))\n",
    "else:\n",
    "    print(\"The cross validated model is not one-calibrated at time {}.\".format(one_cal_times))"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    },
    "ExecuteTime": {
     "end_time": "2023-06-08T04:41:54.437576645Z",
     "start_time": "2023-06-08T04:41:54.038452310Z"
    }
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Run d-calibration test at 50% time for the whole data set."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 0 Axes>"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": "<Figure size 640x480 with 1 Axes>",
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjIAAAG7CAYAAADZm3/UAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABDCElEQVR4nO3deXwV9f398XNDQlgTFjUJAmELm5hQWaOAiiwqqEhwqVpFod+KgAIKFRQRRFFRFimIrRaqFKVRWdxwoYKogBoqCihIZYeETRIWs79/f/C7IxcC5Ibc3Dv09Xw88pDMnUxOxpl7z5078xmPmZkAAABcKCzYAQAAAEqKIgMAAFyLIgMAAFyLIgMAAFyLIgMAAFyLIgMAAFyLIgMAAFyLIgMAAFwrPNgBAq2wsFC7du1S1apV5fF4gh0HAAAUg5np0KFDqlWrlsLCTn3c5ZwvMrt27VKdOnWCHQMAAJTA9u3bVbt27VM+fs4XmapVq0o6tiKioqKCnAYAABRHVlaW6tSp47yOn8o5X2S8HydFRUVRZAAAcJkznRbCyb4AAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1KDIAAMC1woMdACiueg+/F+wIJ9nydI9gRwCA/2kckQEAAK5FkQEAAK5FkQEAAK7FOTJngXM2AAAILo7IAAAA16LIAAAA16LIAAAA16LIAAAA16LIAAAA16LIAAAA16LIAAAA16LIAAAA16LIAAAA12JkXwDAWQm1Uc6LO8I5uUtHsEeUp8gAKFKoPVlKwX/CBBB6+GgJAAC4FkdkACBEhNpRMI6AwQ04IgMAAFyLIgMAAFyLj5aAAAu1jwskPjIAcO7giAwAAHAtigwAAHAtigwAAHAtigwAAHAtigwAAHAtigwAAHAtigwAAHAtxpEBcM4JtbF7GLcHCByOyAAAANeiyAAAANeiyAAAANeiyAAAANeiyAAAANeiyAAAANeiyAAAANeiyAAAANeiyAAAANcKmSLz9NNPy+PxaMiQIc607OxsDRw4UDVr1lSVKlWUkpKijIyM4IUEAAAhJSSKzNdff62XXnpJiYmJPtOHDh2qd955R6mpqVq2bJl27dql3r17ByklAAAINUEvMocPH9btt9+uv/3tb6pevbozPTMzU6+88oomTZqkzp07q1WrVpo1a5a+/PJLrVy5MoiJAQBAqAh6kRk4cKB69OihLl26+ExPS0tTXl6ez/SmTZuqbt26WrFixSmXl5OTo6ysLJ8vAABwbgrq3a/feOMNrV69Wl9//fVJj6Wnp6t8+fKqVq2az/SYmBilp6efcpkTJkzQ2LFjSzsqAAAIQUE7IrN9+3Y98MAD+uc//6kKFSqU2nJHjhypzMxM52v79u2ltmwAABBaglZk0tLStGfPHl1yySUKDw9XeHi4li1bphdeeEHh4eGKiYlRbm6uDh486PNzGRkZio2NPeVyIyMjFRUV5fMFAADOTUH7aOmqq67S999/7zPt7rvvVtOmTfXnP/9ZderUUUREhJYsWaKUlBRJ0oYNG7Rt2zYlJycHI/I5o97D7wU7wkm2PN0j2BEAAC4UtCJTtWpVtWjRwmda5cqVVbNmTWd6v379NGzYMNWoUUNRUVEaPHiwkpOT1b59+2BEBgAAISaoJ/ueyeTJkxUWFqaUlBTl5OSoe/fumjFjRrBjAQCAEBFSRWbp0qU+31eoUEHTp0/X9OnTgxMIAACEtKCPIwMAAFBSFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaQS0yL774ohITExUVFaWoqCglJyfrgw8+cB7Pzs7WwIEDVbNmTVWpUkUpKSnKyMgIYmIAABBKglpkateuraefflppaWn65ptv1LlzZ91www1at26dJGno0KF65513lJqaqmXLlmnXrl3q3bt3MCMDAIAQEh7MX37dddf5fP/kk0/qxRdf1MqVK1W7dm298sormjt3rjp37ixJmjVrlpo1a6aVK1eqffv2wYgMAABCSMicI1NQUKA33nhDR44cUXJystLS0pSXl6cuXbo48zRt2lR169bVihUrTrmcnJwcZWVl+XwBAIBzU9CLzPfff68qVaooMjJS9957r+bPn6/mzZsrPT1d5cuXV7Vq1Xzmj4mJUXp6+imXN2HCBEVHRztfderUCfBfAAAAgsXvIrN69Wp9//33zvcLFy5Ur169NGrUKOXm5vodoEmTJvr222+1atUqDRgwQHfddZfWr1/v93K8Ro4cqczMTOdr+/btJV4WAAAIbX4XmT/96U/auHGjJOnnn3/WrbfeqkqVKik1NVUjRozwO0D58uXVqFEjtWrVShMmTFBSUpKmTp2q2NhY5ebm6uDBgz7zZ2RkKDY29pTLi4yMdK6C8n4BAIBzk99FZuPGjWrZsqUkKTU1VZ06ddLcuXM1e/ZsvfXWW2cdqLCwUDk5OWrVqpUiIiK0ZMkS57ENGzZo27ZtSk5OPuvfAwAA3M/vq5bMTIWFhZKkTz75RD179pQk1alTR/v27fNrWSNHjtQ111yjunXr6tChQ5o7d66WLl2qDz/8UNHR0erXr5+GDRumGjVqKCoqSoMHD1ZycjJXLAEAAEklKDKtW7fW+PHj1aVLFy1btkwvvviiJGnz5s2KiYnxa1l79uzRnXfeqd27dys6OlqJiYn68MMP1bVrV0nS5MmTFRYWppSUFOXk5Kh79+6aMWOGv5EBAMA5yu8iM2XKFN1+++1asGCBHnnkETVq1EiS9Oabb+rSSy/1a1mvvPLKaR+vUKGCpk+frunTp/sbEwAA/A/wu8gkJib6XLXkNXHiRJUrV65UQgEAABRHiUf2zc3N1Z49e5zzZbzq1q171qEAAACKw+8is3HjRvXr109ffvmlz3Qzk8fjUUFBQamFAwAAOB2/i8zdd9+t8PBwvfvuu4qLi5PH4wlELgAAgDPyu8h8++23SktLU9OmTQORBwAAoNj8HhCvefPmfo8XAwAAEAh+F5lnnnlGI0aM0NKlS7V//37uNA0AAILG74+WunTpIkm66qqrfKZzsi8AAChrfheZTz/9NBA5AAAA/OZ3kbn88ssDkQMAAMBvxSoy3333nVq0aKGwsDB99913p503MTGxVIIBAACcSbGKTMuWLZWenq4LLrhALVu2lMfjkZmdNB/nyAAAgLJUrCKzefNmnX/++c6/AQAAQkGxikx8fHyR/wYAAAimEt00csOGDZo2bZp++OEHSVKzZs00ePBgNWnSpFTDAQAAnI7fA+K99dZbatGihdLS0pSUlKSkpCStXr1aLVq00FtvvRWIjAAAAEXy+4jMiBEjNHLkSI0bN85n+pgxYzRixAilpKSUWjgAAIDT8fuIzO7du3XnnXeeNP2OO+7Q7t27SyUUAABAcfhdZK644gotX778pOmff/65OnbsWCqhAAAAiqNYHy0tWrTI+ff111+vP//5z0pLS1P79u0lSStXrlRqaqrGjh0bmJQAAABFKFaR6dWr10nTZsyYoRkzZvhMGzhwoO69995SCQYAAHAmxSoyhYWFgc4BAADgN7/PkQEAAAgVFBkAAOBaFBkAAOBaFBkAAOBaFBkAAOBaJbppZGFhoTZt2qQ9e/acdEVTp06dSiUYAADAmfhdZFauXKnbbrtNW7dulZn5PObxeFRQUFBq4QAAAE7H7yJz7733qnXr1nrvvfcUFxcnj8cTiFwAAABn5HeR+emnn/Tmm2+qUaNGgcgDAABQbH6f7NuuXTtt2rQpEFkAAAD84vcRmcGDB+vBBx9Uenq6Lr74YkVERPg8npiYWGrhAAAATsfvIpOSkiJJuueee5xpHo9HZsbJvgAAoEz5XWQ2b94ciBwAAAB+87vIxMfHByIHAACA34pVZBYtWqRrrrlGERERWrRo0Wnnvf7660slGAAAwJkUq8j06tVL6enpuuCCC9SrV69Tzsc5MgAAoCwVq8gcfxuCE29JAAAAECzcNBIAALgWRQYAALgWRQYAALgWRQYAALgWRQYAALiW30Vm9erV+v77753vFy5cqF69emnUqFHKzc0t1XAAAACn43eR+dOf/qSNGzdKkn7++WfdeuutqlSpklJTUzVixIhSDwgAAHAqfheZjRs3qmXLlpKk1NRUderUSXPnztXs2bP11ltvlXY+AACAU/K7yJiZMyjeJ598omuvvVaSVKdOHe3bt6900wEAAJyG30WmdevWGj9+vF577TUtW7ZMPXr0kHTsrtgxMTGlHhAAAOBU/C4ykydP1urVqzVo0CA98sgjatSokSTpzTff1KWXXlrqAQEAAE6lWPdaOl5SUpLPVUteEydOVHi434sDAAAoMb+PyDRo0ED79+8/aXp2drYaN25cKqEAAACKw+8is2XLFhUUFJw0PScnRzt27CiVUAAAAMVR7M+CFi1a5Pz7ww8/VHR0tPN9QUGBlixZovr165duOgAAgNModpHp1auXJMnj8eiuu+7yeSwiIkL16tXT888/X6rhAAAATqfYRcY7dkz9+vX19ddf67zzzgtYKAAAgOLw+zKjzZs3ByIHAACA30p0vfSSJUu0ZMkS7dmzxzlS4/X3v/+9VIIBAACcid9FZuzYsRo3bpxat26tuLg4eTyeQOQCAAA4I7+LzMyZMzV79mz94Q9/CEQeAACAYvN7HJnc3FxuRQAAAEKC30Wmf//+mjt3biCyAAAA+MXvj5ays7P117/+VZ988okSExMVERHh8/ikSZNKLRwAAMDp+F1kvvvuO7Vs2VKStHbtWp/HOPEXAACUJb+LzKeffhqIHAAAAH7z+xwZAACAUOH3EZkrr7zytB8h/fvf/z6rQAAAAMXl9xGZli1bKikpyflq3ry5cnNztXr1al188cV+LWvChAlq06aNqlatqgsuuEC9evXShg0bfObJzs7WwIEDVbNmTVWpUkUpKSnKyMjwNzYAADgH+X1EZvLkyUVOf/zxx3X48GG/lrVs2TINHDhQbdq0UX5+vkaNGqVu3bpp/fr1qly5siRp6NCheu+995Samqro6GgNGjRIvXv31hdffOFvdAAAcI4p0b2WinLHHXeobdu2eu6554r9M4sXL/b5fvbs2brggguUlpamTp06KTMzU6+88ormzp2rzp07S5JmzZqlZs2aaeXKlWrfvn1pxQcAAC5Uaif7rlixQhUqVDirZWRmZkqSatSoIUlKS0tTXl6eunTp4szTtGlT1a1bVytWrChyGTk5OcrKyvL5AgAA5ya/j8j07t3b53sz0+7du/XNN99o9OjRJQ5SWFioIUOG6LLLLlOLFi0kSenp6SpfvryqVavmM29MTIzS09OLXM6ECRM0duzYEucAAADu4XeRiY6O9vk+LCxMTZo00bhx49StW7cSBxk4cKDWrl2rzz//vMTLkKSRI0dq2LBhzvdZWVmqU6fOWS0TAACEJr+LzKxZs0o9xKBBg/Tuu+/qs88+U+3atZ3psbGxys3N1cGDB32OymRkZCg2NrbIZUVGRioyMrLUMwIAgNBT4pN909LS9MMPP0iSLrroIv3ud7/zexlmpsGDB2v+/PlaunSp6tev7/N4q1atFBERoSVLliglJUWStGHDBm3btk3JyckljQ4AAM4RfheZPXv26NZbb9XSpUudoyQHDx7UlVdeqTfeeEPnn39+sZc1cOBAzZ07VwsXLlTVqlWd816io6NVsWJFRUdHq1+/fho2bJhq1KihqKgoDR48WMnJyVyxBAAA/L9qafDgwTp06JDWrVunAwcO6MCBA1q7dq2ysrJ0//33+7WsF198UZmZmbriiisUFxfnfM2bN8+ZZ/LkyerZs6dSUlLUqVMnxcbG6u233/Y3NgAAOAf5fURm8eLF+uSTT9SsWTNnWvPmzTV9+nS/T/Y1szPOU6FCBU2fPl3Tp0/3NyoAADjH+X1EprCwUBERESdNj4iIUGFhYamEAgAAKA6/i0znzp31wAMPaNeuXc60nTt3aujQobrqqqtKNRwAAMDp+F1k/vKXvygrK0v16tVTw4YN1bBhQ9WvX19ZWVmaNm1aIDICAAAUye9zZOrUqaPVq1frk08+0Y8//ihJatasmc9tBAAAAMpCicaR8Xg86tq1q7p27VraeQAAAIqt2B8t/fvf/1bz5s2LvAljZmamLrroIi1fvrxUwwEAAJxOsYvMlClT9Mc//lFRUVEnPRYdHa0//elPmjRpUqmGAwAAOJ1iF5k1a9bo6quvPuXj3bp1U1paWqmEAgAAKI5iF5mMjIwix4/xCg8P1969e0slFAAAQHEUu8hceOGFWrt27Skf/+677xQXF1cqoQAAAIqj2EXm2muv1ejRo5WdnX3SY7/++qvGjBmjnj17lmo4AACA0yn25dePPvqo3n77bTVu3FiDBg1SkyZNJEk//vijpk+froKCAj3yyCMBCwoAAHCiYheZmJgYffnllxowYIBGjhzp3PDR4/Goe/fumj59umJiYgIWFAAA4ER+DYgXHx+v999/X7/88os2bdokM1NCQoKqV68eqHwAAACnVKKRfatXr642bdqUdhYAAAC/+H3TSAAAgFBBkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK5FkQEAAK4V1CLz2Wef6brrrlOtWrXk8Xi0YMECn8fNTI899pji4uJUsWJFdenSRT/99FNwwgIAgJAT1CJz5MgRJSUlafr06UU+/uyzz+qFF17QzJkztWrVKlWuXFndu3dXdnZ2GScFAAChKDyYv/yaa67RNddcU+RjZqYpU6bo0Ucf1Q033CBJevXVVxUTE6MFCxbo1ltvLcuoAAAgBIXsOTKbN29Wenq6unTp4kyLjo5Wu3bttGLFiiAmAwAAoSKoR2ROJz09XZIUExPjMz0mJsZ5rCg5OTnKyclxvs/KygpMQAAAEHQhe0SmpCZMmKDo6Gjnq06dOsGOBAAAAiRki0xsbKwkKSMjw2d6RkaG81hRRo4cqczMTOdr+/btAc0JAACCJ2SLTP369RUbG6slS5Y407KysrRq1SolJyef8uciIyMVFRXl8wUAAM5NQT1H5vDhw9q0aZPz/ebNm/Xtt9+qRo0aqlu3roYMGaLx48crISFB9evX1+jRo1WrVi316tUreKEBAEDICGqR+eabb3TllVc63w8bNkySdNddd2n27NkaMWKEjhw5ov/7v//TwYMH1aFDBy1evFgVKlQIVmQAABBCglpkrrjiCpnZKR/3eDwaN26cxo0bV4apAACAW4TsOTIAAABnQpEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACuRZEBAACu5YoiM336dNWrV08VKlRQu3bt9NVXXwU7EgAACAEhX2TmzZunYcOGacyYMVq9erWSkpLUvXt37dmzJ9jRAABAkIV8kZk0aZL++Mc/6u6771bz5s01c+ZMVapUSX//+9+DHQ0AAARZeLADnE5ubq7S0tI0cuRIZ1pYWJi6dOmiFStWFPkzOTk5ysnJcb7PzMyUJGVlZZV6vsKco6W+zLNVnL+T3KWH3GWruPtxqGUnd9kid9kKxOvr8cs1s9PPaCFs586dJsm+/PJLn+nDhw+3tm3bFvkzY8aMMUl88cUXX3zxxdc58LV9+/bTdoWQPiJTEiNHjtSwYcOc7wsLC3XgwAHVrFlTHo8niMlOLSsrS3Xq1NH27dsVFRUV7DjFRu6yRe6yRe6yRe6y5YbcZqZDhw6pVq1ap50vpIvMeeedp3LlyikjI8NnekZGhmJjY4v8mcjISEVGRvpMq1atWqAilqqoqKiQ3aBOh9xli9xli9xli9xlK9RzR0dHn3GekD7Zt3z58mrVqpWWLFniTCssLNSSJUuUnJwcxGQAACAUhPQRGUkaNmyY7rrrLrVu3Vpt27bVlClTdOTIEd19993BjgYAAIIs5IvMLbfcor179+qxxx5Tenq6WrZsqcWLFysmJibY0UpNZGSkxowZc9JHYqGO3GWL3GWL3GWL3GXLrbmL4jE703VNAAAAoSmkz5EBAAA4HYoMAABwLYoMAABwLYoMAABwLYrMOeRcOG977dq1wY5QYjt27Ah2hBIhd9lx6/bt1tySu7OH8nP65s2bNW7cOC1fvjzYUSgybuXdwGfPnu3cCTxUb8FQXC+//LLuvfdeDRs2rPg3CwsRjz/+uO6++2499dRTwY7iF3KXHbdu327NLbk3+2effSYpdJ/T//a3v6lhw4Z6/PHH9f777+vgwYPBDXSW93VEkE2dOtXat29vt956q61duzbYcfxSWFhoZmYFBQXOf3/88Udr3bq13XzzzbZu3Tqf+UKJN1N+fr6ZmWVmZtrnn39uderUsQcffNB+/vlnM/vtbwsV3jxuy+3N66bcJ2bJy8tzxfbtXcdeOTk5rshtdvJ+mZ+f75rsXosXL7Y2bdpYv3797IcffjCz0Mv7wQcfWHp6uk2bNs2aNm1q7733XlDzUGRc5PiN+fgnyZ07d9o111xjXbt2tVWrVp00b6g51YuN98ln1apVdvPNN1vz5s0tKyurLKOd0amye9f3ggULrGfPnnbllVeWZSy/5OXlnfTvUM19/PrOzs62X3/91cx+21ZCMffxmXNycuybb77xmRaq2/eJ6/rhhx+2Q4cOOdNCNbeZb/ai9tFQzn7iGzozs127dtmNN95ol112ma1cuTJY0czM97WkqNeVSy65xO68807btm1bWcbyQZFxgeM38ONfhI5/bM2aNXb77bdbo0aNLDc3t0zzldTs2bPt+eeft9TUVPvll198Htu9e7fVrl3b/u///s/nyTSYjt+J//a3v9nDDz9sL7/8snM0wGvFihVWvXp1Gzt2bFlHPKN//vOfVqNGDXvttdfMzHy2lVDLffz6fvbZZ83j8dirr7560mOhltvr+eeft/PPP9/uv/9+++mnn3weC6Xtu7Cw0Gd9TpgwwapXr24ej8fZTrxCKXdRpk2bZjfccIM99NBDlpqa6vNYqGU/1Zsi7/StW7faH//4R4uLi7PvvvuuLKP55DA7VshP5H0teuutt6x27do2e/bsMst2IoqMizz55JM2dOhQ27Vrl5md3I5//vlnu/DCC23o0KHBiFdsX3zxhTVv3twaNWpkvXv3tsjISOvXr5/zd3l3oPfff9/CwsLsgw8+CGZcH59++qk1bdrUGjVqZH379rXzzjvPbr75ZktPT/eZb+bMmVahQgXbsGFDkJIW7bbbbrOGDRtahw4d7PDhw2bm+1FCqOX+4IMPLD4+3uLj423evHk+jx2//YdK7sLCQsvOzrYBAwZYQkKCvfbaa/bLL784R5IKCgqc3KG2fS9atMjq1atnDRs2tL/+9a/WqFEjp8gc/6IWarnNzDZu3GiXXHKJNWnSxEaPHm1XXHGFXXDBBc5HM6H8nDJjxgwbOnSovfLKK7Zx40Yz8922ExMTLSUl5aQ3TGVl/Pjx1qdPH/vxxx/NrOgCdt1111n37t3t+++/L+t4ZkaRcYUffvjBunbtahUqVLDatWvbW2+9dcp558yZYxEREbZly5YyTFh8aWlp1qlTJ3vooYfs6NGjZnbsKEHbtm3t7bffdubz7shdu3a1nj17mlnwz39Yvny5JSUl2WOPPea8MM2ePdsaN27sfPbulZeXZ7/73e+sX79+Zhb87F6tW7e2oUOHWocOHWzMmDFm5ltkQin3Rx99ZB6Pxx588EFnmnebOVEo5d68ebMlJSXZF198YWZm+/btsy1btvgcTfXmC4XtOz8/35544gkLDw+3p556ytkeLrvsMuvbt6+Z/bY/huJ+aWb2+OOP24033mgHDx40M7N169ZZYmKivfnmm848oZb9p59+sksuucQaNWpk999/vzVr1swuuugi++qrr8zst/3yyy+/tJiYGJs5c+ZJ5y8FOl+PHj2sevXq1qBBA3v++edPmseb59tvv7ULL7zQnnnmGee5sSxx1VKIMzNt2rRJUVFRWr58uRo2bKi5c+fqv//9r/P48Xr37q3mzZtr8uTJkqTCwsIyz3w6O3fuVKtWrTRw4EBVqFBBkpSSkqItW7YoLy/Pmc/7dw0fPlwfffSRduzYobCw4G6uF1xwgR5//HENGTLEyd6wYUM1b95ctWvXduYrLCxUeHi4Bg4cqI8++kh79+4NenYzU0ZGhmrXrq3+/fsrOTlZ//rXv7Rp0yaVK1dOBQUFkhRSuZs2barExERnGx41apT69OmjXr16ac6cOdq3b58kqaCgIKRyL168WJGRkbr00kv10EMPqWPHjrr22mvVrVs3LV68WNJv+2UobN/lypVT9+7dtWfPHo0cOVLlypXTr7/+qiZNmmjPnj06cuSIc/VMKO6XkvTxxx+rYsWKio6OliRVqlRJ1atXV5s2bZznlfz8fEmhk33evHlq3LixVq9eralTp+qDDz5QgwYNdM8992jnzp0qV66czEzJycnq0qWLXn31VZ/nyEDbv3+/ypUrp0WLFqljx45avHixVq5cKem37cCbMSkpSb1799a8efP0zTfflFlGr+BvgTgtj8ejSy65RKNHj1br1q01btw4rVy5Uh999JHy8/NPujwvIiJCt912m9LS0nTgwIGQeJI5XqdOnTRu3DjVq1fPyZ6Xl6cLLrhANWrUcObz5m7UqJE6deqkjz/+OCh5j9ewYUNdd911ql69uiRp5syZ6tq1qzZu3KiuXbvqueeek/Rb9saNGyshIUHff/990DJ7eTweVahQQT/99JPOP/983XDDDapdu7YmTZqk1NRUrVq1ynlyCpXcderU0ejRozVlyhTFx8dr1apVateunfLy8jR27Fg9+OCDko49mUrBz+1df3Fxcdq/f78mTpyo//znP5oxY4YeffRR1a5dW3369NG2bdsUHh4uKfjbtzdzmzZtVL16ddmxo/SqWLGiqlevrh07dqhy5crOfKG4X+bm5urKK6/Uu+++q3/+85+aPHmyEhMTtWXLFl177bXq2bOnfvnlF0VEREg6th8HO3teXp4WLlyo6OhoVa1aVXl5eYqPj1dcXJzWrVunl156SQUFBU7hHT16tFauXHnKN7CBkJiYqPHjx6tDhw4aMmSIDhw4oPnz5ys7O1sej8fJ4P3vE088oaysLL399tvav39/wPMdL7Re5VCkWrVqKSkpSdKxInDFFVfo9ddf15o1a06aNzw8XHXq1HGeKMtScY7+REdHq0qVKs4TpnTsKE1GRoYaNWp00vxxcXHKyclxbjUfqCNMxVluuXLlnBfNFStW6IknntDEiRP16quv6tprr9X06dP19NNPO/O3bNlSmzZtcr4PxJOPP+tj/fr1KleunM4//3xddtllSkhI0D/+8Q/dcsstKleunPOONZRyX3XVVbr//vvVu3dvLVy4UI899pjee+89DRkyRKtXr9a8efOceQOd+0yZvcU8MjJSsbGxmjZtmq6++mpdccUV+v3vf6+XXnpJ9erV89lGymL7Pt0yT3wjdPwLVOfOnbV582b9/PPPJ80XSvtl+fLldd9992nYsGFavny5Ro8erQkTJujLL7/UtGnTtGXLFg0ZMsSZv1atWmWS/XRycnJUv359SVJWVpZTssxMN954oyZNmqRffvlF5cqVU2FhoeLj49WlSxelpqZKKpvxZSpWrKiLL75Y0rF9q1u3bvr888/14Ycf+swXFhamgoICRUdHa9CgQVq0aJE+//zzgOfzyVCmvw0OM3MO5584/VS8848fP15bt27VokWLdOjQoZPmu+aaa/Ttt9/qwIEDpRf4NI5/t5aVlaVdu3YV+fjxPB6PszMuWLBADRs2VHx8/Ek/U6FCBTVo0EBfffWV8zsClf3w4cNFrs+iJCcna+PGjRo0aJBatWqlxx57TJdeeqm+/vpr5eXlqbCwUFWrVlXbtm21evVqSaX75FOS3FFRUWrdurU2bNiglJQUvfzyy7rwwgt1+eWXq127doqIiFBubm5I5a5WrZqGDh2qESNGqEqVKs6LTkpKigoLC51tPD8/P2C5/c3cqVMn1ahRQzt27FBiYqKTr2LFirrpppuUlpamnJwcmVlAt++S7JcnZoiNjdXWrVt9Hi8oKAi5/TIuLk5jxoxRxYoVddttt2ngwIGKi4vTlVdeqSFDhmjx4sU6cuSIpMA/pxSnGFWpUkWXXXaZvvnmG9100016+eWX1bRpUy1fvlyPPvqoqlWrpjlz5jj5wsPDVbt2bUVFRRX7d5QW7+vO4MGDnY+aMjIyity/HnjgAcXHx+sf//iHz5uKQKPIBInH41G5cuX0008/acaMGVq6dKnPR0VFPcl4z2WoV6+e+vbtqzfffNPZGf/9738rMzNTknT06FFdd911xX5RLo2/RTp2DkOTJk3Uo0cP3XLLLcUeunrVqlW69tpr5fF4tH//fg0aNEhTp06VdGw9dO3aVdddd11Ajgwcn71Fixa65pprNGjQIGcnPNXvNDNVrlzZ+f7IkSNav369mjZtqoiICIWFhSkvL0+tW7fW5ZdfHhK5v/rqK82aNUu/+93vdOTIEX3zzTd6/PHHtXfvXk2ZMkXSsXe3oZbbe8j9+J/Pzc3V9u3bValSJUnHjkQGKrc/mQsLC1WpUiUNGDBAF154oSZNmuTkk44Nl9+qVSvnaEAgt++S7pfeHMnJydq+fbvzvOJ9QfOeFxFK+6X3+8zMTGe/9C7ju+++U7t27RQZGekcCQ5E9uIWR+9/7733Xo0aNUqVK1fWtGnTdMstt+iHH37Q7373O9WrV8/Ztr3ngFWpUkWrVq1yfkdZ8R4ViouL02233aY1a9bovffekyS98847zrk00rHXnh9//FGbN292jvCWidI6axj+KSwstJEjR1pkZKS1bdvWqlatal27drVly5ad8efMjl2l0bJlS7vhhhusXbt2Vq1aNVu6dKmZHRvM6h//+IdzeW1ZmDBhgjVq1MjeeecdmzNnjl1xxRWWkJBgq1evNrOTRwv1ys7OtrZt29r7779v06dPt4oVK9qll17qc9XV1q1bLTs7OyC5CwoKbPjw4da0aVObN2+eTZ482RISEqxz587OpZtnulIgJyfHxo0bZ23atLFvvvnGWa7ZsTP/A3EWf0ly//rrr3brrbfa22+/7Ywfs3PnTrv99tvtoYcessLCwpDMfaK8vDwbPny4de7c2fbu3Wtmv+0Xgchd0szPPfec1ahRw2677TZbuHChPf/889aoUSPn6jxv5m3btgVs+y7pfllYWGg5OTnWqVMnGzx4cJHzBDJ3Sdd5//797fLLL7cZM2bY1q1b7bnnnrN69erZX//6V+fvCnT2kSNHWmxsrLVs2dJuvvlm++yzz3x+94n/zs3N9fn+p59+smrVqtm//vUvn+UuXbrU5s+f7/eVS6VxZdbxIybfeOON1qlTJ2vdurV5PB5nVN/8/Hx77rnnbODAgbZ169az/p3+oMgEyerVqy0xMdE+/vhjy8nJsQ0bNlj79u2ta9euzq0GTvckc/jwYevYsaN5PB67+eabbc+ePT7zlOVlhYcOHbLExER7+umnnWkbNmyw66+/3i6++OIic3n/u3TpUvN4PFa5cmWLi4uz+fPnnzRvIO3du9fq1atns2bNcqYtXbrUunXrZldffbUz7fgX+cLCQsvNzbVFixbZ5MmTrUmTJpaQkGBLliwJeN6zyW128oCKZscuDy4rZ7O+58+fbxMnTrSEhARr2rSp8wIRipnNjl0q/t5771m7du3skksusebNm5fpUO4l2S9PfIFNSEiwkSNHlvkgmyVd599//7317dvXqlWrZhdffLElJCSU6TovaXE0O/b/Kzc31x544AHr1q3bSYP2paen+12+jv//+emnn9rGjRudS9RL8vx68OBB69atm3k8Hvv9739/0uvOqYZHCDSKTJC89tprVqNGDZ8N4aOPPrLOnTvbHXfc4UwramPbtm2bValSxRo3bmxpaWnO9LIcY+B4+fn5lpSUZM8884wzrbCw0FauXGmVK1e26dOn++TLyclxdrC5c+dakyZNbOrUqSctsyzs3r3bmjVr5jOCaUFBgb3++ut23nnn2RtvvOFMM/N99zR37lz7/e9/b1OmTPFZZlncHuJscp8qX1kUx7PJPWvWLLvhhhuKHM8ilDIfv317p+/YscNnmWWxjZzNfukdyXXNmjUBz1mUs1nnR44csXXr1tnnn3/us8xAb98lKY5eu3btsuHDh1tCQoI1aNCgVEv68uXLrXHjxpaQkGAXX3yxtWvXztavX+88Xtxt8eDBg87rztdff+1Mz8vLC/otcSgyQTJ9+nRr27btSaM1Pvfcc9aiRQtbvHixM+3gwYP27rvv+rybPn5DLygoCOrATgcPHrTrr7/e+vfvbwcOHHCmZ2dn24MPPmh169Z1pm3fvt1uvPFGZ/ThgwcP+rzLKOqIQSClp6dby5Yt7YknnvB5N7F7927r27evXX755c60jRs32i233GLPPfecmR37+04cTC6Uc0+aNMnMgnsfrrNZ37/++muR94kKxcynWtdl+WajJPvlQw89ZGYnv9CW9ZukkqzzUxXcstpOzqY4FhQU2KRJk2zatGmlmmnr1q3WqlUre+SRRywrK8u++uoru+6666xx48a2YsWKYi/Huz147+XnnRYKgyGaMSBembP/f6JXly5dlJaWpv/85z8+j3fv3l2xsbH697//7Ux76aWXdO+992rBggXOtI4dO0o6djVEWFhY0MaLMTNFR0erXbt2+vrrr52Tj6Vjl6F269ZNlStX1pIlSyT9NlDV3r17nStNIiMjVVBQIDMr08vGzUwxMTG67LLL9Prrr/tcnREbG6t27drp8OHDzrgkFStW1L59+7Rr1y4VFBSofPnyzolwksose0lz79ixQwUFBWVy6WZp5vau78jISIWHh5fp+i7tde09KbIscpdkv0xPT3eeU4KR25u9JOt8586dRV4JWlb75eHDhxUfH6+ffvpJv/zyi6RjJxy3bNlS9957r5555hlJx9bljh07dOutt2r48OGSjp28O3ToUA0aNEiSSu1E2XfffVdHjhzR8OHDValSJbVp00ZPPfWUNm3apL/+9a/atm2bpJNPRD6Rd3to27atky+YrzsnCVKB+p/mbbF9+vSxNm3a+Lxb8k6/6aabnO9//vlne+yxx066sWIoOP5w7kUXXWT9+/f3OcqUlpZmlStX9jkUuX///jLPWRTv/4esrCyrVKmSjRo1yifbp59+auXLl/e5q+vOnTvLPOeJyH12/DkiFSqZj89SHKG2X/pzRCeU1nlxedf3k08+aUlJST5H1M3MPvzwQ2vWrJl98sknZnZsXd9zzz12xx13BPSI0SuvvGKxsbE+0959912Lj4+3Bg0a2Ouvv37S33Cq70NZiNQpdzM/L+HzttgJEybou+++08yZM3X06FHn8cqVKzvjY5iZ6tevr7Fjx6patWoBHdHR+07Xn9/h8XhUUFCgSpUq6eGHH9bXX3/t3B5Bknbs2KGEhASdf/75zjTvCL5FvXsqqZKMqxAWFuYcFXryySc1d+5cZ+wGSfrvf/+r1q1bO5dBSscG0yrp7yuKW3OXZDsMdm7vMo4f9C3UMx+/nLCwsGIvM9T2S++Ai9u3bz/jz4TCOi+pIUOGKD8/X2+++aY2b97sTD/vvPO0bds25xYKNWrU0MSJE/Xaa6+V+IhRcbbh+Ph41ahRQ/3791d6erpWrlyphx56SA8//LAqVKigTz75RJKco4fr1q3Tiy++qF9//TVoR25LJIglyvWOf4fk72fI3rY7adIki4uLsxEjRtjGjRttxYoV1qRJE/vHP/5x2t9X2o5v396z5UvyufjkyZMtPj7eWrRoYX369LEKFSrYI488ErDshYWFPtlXr15tu3bt8rnb8Jl+3uv++++3+vXrW4cOHaxv375WsWJFe/LJJ8sk98cff2xpaWnOSaFnWvfByn2iF1980WbMmFHs+YOR+8R1PWHCBPv9739f7EtEQ2Vdv/DCC9a2bdsS3bk52PvlX/7yF/N4PDZgwAAzC+3tu6TrxPs3vfbaa5aUlORz2frChQutZcuWRd7M92wupz7dla1mx47IvfHGG1azZk1r0KCBVaxY0Tk/8S9/+YslJCT4/Fz//v2tXr16dujQIVcdkaHIlILx48fbXXfdZS+88IJ9++23ZubfzjBp0iRr3LixtWjRwipXrmwDBgwIyhVIX3zxhTVs2NCuuuoqv3/2+L933bp1NnXqVLvvvvv8OqHsbHzyySd20UUXWVJSktWvX99uv/12y8rKOilbUbyP//rrr7Zs2TJ79NFH7Q9/+IN9+eWXZZK7adOm1qRJE2vSpInVr1/fMjIyQj632bETzj0ej1WrVs1+/PHHYv9csHJ/+OGHVr9+fTv//PNtwYIFfv1sMNf1li1brF27dhYfH2/Tp093xioqjmDvl4sXL7a6devahRdeaK1atbLOnTsX+6OUYKzz41+8582bZ3PmzCnR7yuL4ujP68727dtt2bJlPidO9+3b13r27Gk5OTnO/5Nt27ZZXFxc0D+q8xdF5ixs2bLFLrnkErv44ottwIAB1rBhQ6tdu7atW7euWD9//Ea3b98+++KLL3zeJZZVIy4oKLA5c+ZYy5YtrUOHDlatWjVbtGiRmZXOGf+BPLs9Ly/PnnvuOYuPj7cxY8bYf//7X5syZYq1atXKHnjggbNefqCy5+Xl2YgRIyw+Pt4ef/xx27Nnj33++eeWlJRkf/zjH896+YFc597tcuHChXb33XdbYmKi3XLLLaWyvZZ2bu8YNLfffrt5PB579NFHS23ZXoG+emPatGnWp0+fIov52azzQOY+ePCgtW3b1ipVqmTjxo0zM7MxY8ZY27Zt7ddffz3rbSWQ2b/88ktr2rSpNWjQwNq3b28ej8fefPPNYufyClRxPNvXHbNjV3917tzZuZLKa+fOnTZz5kzLzMzkiMy56sSBw1JTUy0xMdF27dplZsfabI8ePSwxMfGsRjbMz88v88vapk2bZmPGjLHVq1fbnXfeac2aNXMeK2qDLu5GHui/Y9++fXbnnXc6l+h6f+fw4cOtT58+ZzVAU6CzP/bYYzZnzhyfaTfeeKPPIGAlyRDo3N7/988//7zdeeedtmrVKgsLC7MPP/zwrJYbyNzdu3e3jh072pEjR8zMbObMmTZ16lR77733nP26JEdBA72uc3NzrWPHjs6lxaNHj7YbbrjB+vXr5zMEw5nGCDpRoHOvX7/e/vznP9vu3budaQsXLrTy5cs7Y2eV9IUykNkzMjIsOTnZRowYYTk5OZafn2+33XabXXbZZWe97JKWr5K+7py4fvPy8uzjjz+2f/7zn9a4cWNLTk52Tph2U2kpCkWmGI4ePWopKSnWrFkzn43q+PEMvBvC/v37rWrVqjZq1ChnUCmzsh8fpbi8uTMzM51bGixdutQuvPBCZzyEE5/gQ+ns9szMTFuyZIllZmb6ZBk4cKD17NnzpPlDYYf1PpkdOnTIZwydmTNnWkxMjN17770njScRKuM1HO+RRx6xF154wczMevToYZ06dTKzk8eaCCbvtrt+/XqLj4+3lJQUi4+Pt3bt2lnbtm0tNjbWOnbs6PMzobCNeB04cMASEhLsnXfesTvvvNMuvfRSGzt2rHXq1MliYmLs5ZdfduY9cV2Hyt/hzbFkyRJr3Lixvfvuu6ecJ9hSU1Otdu3a9v333zvTxo8fb6NHjy5y/kAWx9J43TnRpEmTLDEx0caPH+93nlBGkTmD3Nxc69Gjh1WpUsW5dM47XPe0adMsLi7OeZfnnT5lyhSrXr16kSd2nW4jCxR/nyQyMzPtkUcesfPOO8+5NPzEHXHt2rU2Y8aMgA9J7U/244cr79Kli/Pkc2L2tLQ0e+uttwJ6HpI/ubOysuxPf/qTnXfeeTZy5Ei777777Pzzz7f+/fs725ZZ6OT2rs/+/fs7Hxv88ssvFhERYTExMdapUyefS2MDnftMmb15hw8fbrVq1bIXX3zRDh48aPv377evvvrKatSoYWPGjDGz34pPKKxr7+OXX365tWjRwnr27OnzjvvWW2+1q6++2ucjhVDdL82ODc5WtWpVW7hwoZkFZ788U8ZVq1aZx+OxN954w44ePWqvvvqqVapUyXr06GF33nmnbdy40fmZQBbH0nrdef/9931GTT98+LDPm6dgjQZf2igyxTBr1ixr166dc+Mxr8WLF1urVq2cDeX44bJjYmJs8uTJZnZsY9m3b58lJiY6h9/L4h3I2VxVtWbNGktKSrJ77rmnyMf79esX0LPbS5o9Pz/fsrOzrWnTpqe8uuPqq6+2Jk2amFnp/38oae7ly5f7jJXx8ccfW2RkpM/Q9qGWu3v37rZ27VrLzs62G2+80SIiIqxSpUr2n//8x2e+QOUububjTxqdNWuWc/TOa8KECRYbG+uzvFBa13PmzLHIyEjr0KGDT7H97rvvrFq1aj63EQj0VScl3b69WU537log13lxlul9vG/fvnbZZZdZkyZNrEaNGjZx4kSbNWuWtWzZ0q666iqfsXcCWRzP9nUnLy/PJk6caB6P56RTHfLz80PmKFhpYByZYujbt6/q1q2rhQsX6ttvv3Wmt2nTRs2bN9eCBQu0efNmhYWFqaCgQBEREerUqZPWrFkj6dgYCjVr1pTH49HixYslqUyu0feOV/Pkk0+qX79+mjZtmpPpTOMtNG3aVAMGDFBqaqrWrFmjXbt2aerUqcrIyJAkjRkzRjk5OcrKygrI31LS7OXKldO6det06NAhJScnS5IOHDjg3HZekp544gmFh4dr//79pZ7d39z2/8eC6NChgzOOhyRlZ2erXLly2rVrl/NzoZK7sLDQGefjtttuU/Xq1XXkyBF9/PHHqlmzpt544w2fnwtU7uJmDgsLk5mpQoUK6tu3r6KionyWk5mZqWrVqikjI8MZUTVU1rUktW/fXp07d9bu3bu1d+9eZ3qVKlUkSfv27XOmPfbYYyG5X3o8Hh09elSxsbHKyspSTk6O81igt5Pjxw8qzhg5f//737VgwQJVq1ZNs2fP1kMPPaS+fftq4cKF2rBhg3bv3u3MO2XKFD377LPOyOSl6Wxfd8LDw9W3b199+umnqlu3rs+yy5Ur565xYs6AIlNM9913n3bt2qX58+c7O0aNGjV00003KT8/X2PHjpV0bAMxM23YsEHNmjWTdOzFysx0/fXXKy4urlQHnDqdrVu3qlWrVpo3b54qVaqkqVOnqmfPnlq/fv0Zh5YuX768evTooeTkZF155ZWqV6+eUlNTVblyZUnH/s4xY8aoSpUqARmk72yyz5s3T23btlV0dLSeffZZxcXF6cknn3TWe2FhoQYPHuwzoFawchf1ZJKdna0FCxbolltuUZs2bZyfC5Xc3mIgHdsH5syZo/fff1+XX365Hn74YT377LPasmVLwHP7k/lUT9oHDx7U+vXr1adPH8XFxTmDk4XKupakhg0bauTIkdq7d6+eeOIJffHFF8rPz9dLL72kNm3a6NJLL3XmDdX90sxUqVIl1a1bVz/88IMiIyOdfIHeTopbvrzbiMfj0X/+8x/t27dP1113nZM/IiJChw4dcm4/IAW+OJ7N6450bCC+yy+/vNRzhZygHAdyqfvvv986depkH330kTMtPz/fZs2aZVWqVLE//OEP9sYbb9jIkSMtPj7eli9fbma/HbJcsWKFz6Hh0lZaZ7ebHTtcOXv2bKtcubI1b97c3nnnnYDlLq3shYWFlp+fbz169LAOHTrYxRdfbBdccIG9/fbbPssuzUOqpbXOjxw5Yp9//rm9/vrr1qxZM0tKSnLubO7vFSmBzn38mBMnfkzjvUneidOCnfl4hw8ftuXLlzvrum3btrZhwwYz++0wfais6+M/ylm4cKElJSVZ7dq1rUmTJlavXj379NNPSz1vaWU/nvf7l156yaKiooocpyRQf0NJLldes2aNhYWF2fz58527O48aNcrat29/0q0RAn25cklfd/6XUGT8sGnTJmvfvr3dd999J9336N1337WuXbtaUlKStWrV6qRbyAdSIM5uX7BggdWvX98ef/xxn+mlfXJYaWc/evSo1apVyzwej02YMCFg2Us79/r1650n2aeeeqrUcgY6d1ko7cxr1661vn37Wu3atQM6Qmwg1nVGRoZ99tlnzgmzbspuZvb222/ba6+9FrDcZmdXvrx/08GDB23YsGHm8XjsyiuvtIsuusgaNGjgnHhblkL1dSeUUGT89Oyzz1qbNm18xv84vol7dxbv9ECfUBWos9sPHTrkcwJbIC4fD0T2X375xRYsWOBcSh6I7KWZe8qUKU7GjRs3+qzz0i6NgdpWAikQ6zo3N9fWr1/vc3TUDeu6qOcSt+yXZXEZfiDK17x582zSpEn2yiuvBDT7mYTa606o4RwZP913332qXr263nnnHf3888+SfG/eFRcXJ+m3m3AF+oSqiIgI9enTRxdddJGTJyIiQpKUkJCgWrVq6eWXX5Z07HNUSRowYIDKly+v+fPnSzp2S/Z169ZpyJAhzm3dq1SpoooVKzonsZX0xmZlmX3r1q2qVq2abrjhBlWuXFn5+fkByV6auYcOHaqtW7cqPDxcCQkJPuvc+7OhmPv4bSWQArGuIyIi1KxZM1WqVMlV67qo5xI37Jfbtm0743ltZysvL0833XSTPvzwQ02bNk01a9ZUXl6eJKlVq1bauHGjjh49Ko/Ho7y8PNWoUUNPPPGEXnzxRZ+Tdz/44AO98MILzvc333yzhg4dqnvuucf5u4Ih1F53Qg1Fxk+VK1dWv379tGHDBmenLWonLe0nxtNx89ntpZk9Pj7eZ9nh4eEByx7I3IFc54HcVgKFdV1269qt2QNd0r2lIRDFsThC8XUnlFBkSuDmm29WixYt9MUXX2jmzJkaNWqUjhw5EtRMbj673a3ZyU3mczW3G7MHsnyFwhGOUHzdCRXBqZfngIEDByolJUUff/yx7r77bpUvXz6oea644gp17NhRS5cuVYcOHdS1a1dJ0rXXXqv9+/dr8ODBKiwsVI8ePbRmzRplZmb6XLYZTG7NTm4yn4lbc0vuzH7fffdp2LBhmj9/vhITExUWFuaUr4kTJ2rs2LGaPXu2ypUrp8LCQm3YsEG333678/OhfrlyqL3uhIxgnJhzrnj77bftxx9/DHYMh5vPbndrdnKXHTdmNnNvbjN3Zj/XL1cOtdedUECROce4+ex2t2Ynd9lxY2Yz9+Y2c192N5YvnB2PWQCGf0TQHDlyRL1791b16tX11FNPqUGDBiosLDzpxLCCgoKQOzHMrdnJXXbcmFlyb27JndknTpyo1NRUPfDAA85HR2bmnOuye/du50of70tgKJwHg5LhZN9zjJvPbndrdnKXHTdmltybW3Jndi5X/t/CEZlz1B/+8AcdOXJE3bp107Zt2/TII48490kKdW7NTu6y48bMkntzS+7L/q9//UsTJkzQHXfcoQcffDDYcRBAHJE5Rw0cOFCrVq3S8OHDdfjwYVed3e7W7OQuO27MLLk3t+S+7Fyu/L+DIzLnsPnz56t58+Zq0qRJsKP4za3ZyV123JhZcm9uyX3ZV65cqZSUFGVlZenuu+/W888/7wyUh3MHRQYAcM5yW/mC/ygyAADAtThHBgAAuBZFBgAAuBZFBgAAuBZFBgAAuBZFBgAAuBZFBgAAuBZFBgAAuBZFBgAAuBZFBgAAuNb/AxcWKhZtLBR3AAAAAElFTkSuQmCC"
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The cross validated model is d-calibrated!\n"
     ]
    }
   ],
   "source": [
    "p_value2, bin_statistics = eval.d_calibration()\n",
    "bins = [\"[0, 0.1)\", \"[0.1, 0.2)\", \"[0.2, 0.3)\", \"[0.3, 0.4)\", \"[0.4, 0.5)\", \"[0.5, 0.6)\", \"[0.6, 0.7)\", \"[0.7, 0.8)\",\n",
    "        \"[0.8, 0.9)\", \"[0.9, 1]\"]\n",
    "plt.clf()\n",
    "fig, ax = plt.subplots()\n",
    "ax.bar(bins, bin_statistics)\n",
    "plt.setp(ax.get_xticklabels(), rotation=30)\n",
    "plt.ylabel(\"Counts in bins\")\n",
    "plt.show()\n",
    "if p_value2 >= 0.05:\n",
    "    print(\"The cross validated model is d-calibrated!\")\n",
    "else:\n",
    "    print(\"The cross validated model is not d-calibrated!\")"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    },
    "ExecuteTime": {
     "end_time": "2023-06-08T04:41:56.866304432Z",
     "start_time": "2023-06-08T04:41:56.517264685Z"
    }
   }
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    },
    "ExecuteTime": {
     "end_time": "2023-06-08T04:41:58.065674923Z",
     "start_time": "2023-06-08T04:41:58.060587140Z"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}