notebooks/model_training.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model training, application and evaluation with Jack\n",
    "\n",
    "This notebook explains how Jack can be used programatically to setup, train and evaluate readers. It serves as an example to familiarize yourself with the framework. This is why we're training both models on a very small subset of the dataset (due to slow training). However, in practice we use scripts (`bin/jack-train.py`, `bin/jack-eval.py`) and configs (`conf/`) to train and evaluate models. See Jack documentation for further details.\n",
    "\n",
    "\n",
    "## Prerequisites\n",
    "\n",
    "**Note:** this command needs to be run in terminal from the root of Jack.\n",
    "\n",
    "Download GloVe [[1]](#ref1):\n",
    "> `sh data/GloVe/download_small.sh`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "import os\n",
    "os.chdir('..')    # change dir to Jack root"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/lib64/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6\n",
      "  return f(*args, **kwds)\n"
     ]
    }
   ],
   "source": [
    "from jack import readers\n",
    "from jack.core import SharedResources\n",
    "from jack.eval import evaluate_reader, pretty_print_results\n",
    "from jack.io.embeddings.embeddings import load_embeddings\n",
    "from jack.io.load import load_jack\n",
    "from jack.util.hooks import LossHook\n",
    "from jack.util.vocab import Vocab\n",
    "from notebooks.prettyprint import QAPrettyPrint, print_nli\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's check all the currently available readers from `readers.py`:\n",
    "\n",
    "These are the **extractive question answering readers**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['fastqa_reader', 'modular_qa_reader', 'fastqa_reader_torch']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[r for r in readers.extractive_qa_readers.keys()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...the **natural language inference readers**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['dam_snli_reader', 'cbilstm_nli_reader', 'modular_nli_reader']"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[r for r in readers.nli_readers.keys()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "and the **link prediction readers**:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['distmult_reader', 'complex_reader', 'transe_reader']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[r for r in readers.link_prediction_readers.keys()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Shared resources\n",
    "\n",
    "To train the reader, we need to define a vocabulary. Additionally, our readers will need word embeddings too. We will use the downloaded GloVe [[1]](#ref1) embeddings. Both the vocabulary and the embeddings are shared between the two presented readers in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "glove_path = 'data/GloVe/glove.6B.50d.txt'\n",
    "embeddings = load_embeddings(glove_path, type='glove')\n",
    "vocab = Vocab(vocab=embeddings.vocabulary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## FastQA model (SQuAD dataset)\n",
    "\n",
    "We will be training a FastQA [[2]](#ref2) model on a very small subset of the SQuAD dataset [[3]](#ref3).\n",
    "\n",
    "Additionally, please note a simpler way to train FastQA from the command line:\n",
    "\n",
    "> `python3 bin/jack-train.py with config='./conf/qa/squad/fastqa.yaml'`\n",
    "\n",
    "\n",
    "### Data loading\n",
    "\n",
    "Load up the training data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "squad_path = 'data/SQuAD/snippet.jtr.json'\n",
    "fastqa_train_data = load_jack(squad_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the reader\n",
    "\n",
    "We need to define the hyperparameter values (representation dimensionality, input representation dimensionality, etc.) and general configuration parameters (maximum span size, etc.) for the FastQA reader:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "fastqa_config = {\n",
    "    \"reader\": \"fastqa_reader\",\n",
    "    \"repr_dim\": 10,\n",
    "    \"max_span_size\": 10\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we create an example reader, based on the (previously defined) vocabulary and the reader configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "shared_resources = SharedResources(vocab, fastqa_config, embeddings)\n",
    "fastqa_reader = readers.fastqa_reader(shared_resources)\n",
    "# equivalent: readers.readers[fastqa_config[\"reader\"]](shared_resources)\n",
    "# equivalent: readers.readers[\"fastqa_reader\"](shared_resources)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Afterwards, we set up modules (input, model, output) given a training dataset. `is_training` set to `True` indicates we are in the training phase. After this call, all the parameters of the model will be initialised."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "fastqa_reader.setup_from_data(fastqa_train_data, is_training=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Applying the untrained reader\n",
    "\n",
    "Our model is initialised, but has not been trained yet. We can see that from the predictions it makes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?\n",
      "Answer:   drive \t (score: 0.001)\n",
      "\n",
      "Question: What is in front of the Notre Dame Main Building?\n",
      "Answer:   copper statue of Christ with arms upraised with the \t (score: 0.001)\n",
      "\n",
      "Question: The Basilica of the Sacred heart at Notre Dame is beside to which structure?\n",
      "Answer:   copper statue of \t (score: 0.001)\n",
      "\n",
      "Question: What is the Grotto at Notre Dame?\n",
      "Answer:   in a direct \t (score: 0.001)\n",
      "\n",
      "Question: What sits on top of the Main Building at Notre Dame?\n",
      "Answer:   in a direct \t (score: 0.001)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num_questions = 5\n",
    "questions = [q for q, a in fastqa_train_data]\n",
    "for q, a in zip(questions[:num_questions], fastqa_reader(questions)):\n",
    "    print(\"Question: \" + q.question)\n",
    "    print(\"Answer:   %s \\t (score: %.3f)\\n\" % (a[0].text, a[0].score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Of course the output is not correct because the model was not trained at all.\n",
    "\n",
    "### Training\n",
    "\n",
    "First, we set up everything necessary for training. In this case we set the `batch_size` to the size of the dataset, as we're working on a very small dataset. We define hooks which will print out useful information during training (loss and speed) and define the optimiser used (Adam)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = len(fastqa_train_data)\n",
    "hooks = [LossHook(fastqa_reader, iter_interval=1)]\n",
    "optimizer = tf.train.AdamOptimizer(0.11)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and we start the training procedure:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/usr/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:96: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.\n",
      "  \"Converting sparse IndexedSlices to a dense Tensor of unknown shape. \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:jack.core.reader:Number of parameters: 6341\n",
      "INFO:jack.core.reader:Start training...\n",
      "INFO:jack.util.hooks:Epoch 1\tIter 1\ttrain loss 10.043580055236816\n",
      "INFO:jack.util.hooks:Epoch 2\tIter 2\ttrain loss 9.839237213134766\n",
      "INFO:jack.util.hooks:Epoch 3\tIter 3\ttrain loss 8.879857063293457\n",
      "INFO:jack.util.hooks:Epoch 4\tIter 4\ttrain loss 7.2919135093688965\n",
      "INFO:jack.util.hooks:Epoch 5\tIter 5\ttrain loss 7.520418167114258\n",
      "INFO:jack.util.hooks:Epoch 6\tIter 6\ttrain loss 6.51716423034668\n",
      "INFO:jack.util.hooks:Epoch 7\tIter 7\ttrain loss 5.9194488525390625\n",
      "INFO:jack.util.hooks:Epoch 8\tIter 8\ttrain loss 5.289041519165039\n",
      "INFO:jack.util.hooks:Epoch 9\tIter 9\ttrain loss 4.665318012237549\n",
      "INFO:jack.util.hooks:Epoch 10\tIter 10\ttrain loss 4.36320161819458\n",
      "INFO:jack.util.hooks:Epoch 11\tIter 11\ttrain loss 3.804948091506958\n",
      "INFO:jack.util.hooks:Epoch 12\tIter 12\ttrain loss 4.801826477050781\n",
      "INFO:jack.util.hooks:Epoch 13\tIter 13\ttrain loss 3.42681622505188\n",
      "INFO:jack.util.hooks:Epoch 14\tIter 14\ttrain loss 3.3622875213623047\n",
      "INFO:jack.util.hooks:Epoch 15\tIter 15\ttrain loss 3.2407429218292236\n",
      "INFO:jack.util.hooks:Epoch 16\tIter 16\ttrain loss 2.87150502204895\n",
      "INFO:jack.util.hooks:Epoch 17\tIter 17\ttrain loss 2.6156039237976074\n",
      "INFO:jack.util.hooks:Epoch 18\tIter 18\ttrain loss 2.2696073055267334\n",
      "INFO:jack.util.hooks:Epoch 19\tIter 19\ttrain loss 2.2451469898223877\n",
      "INFO:jack.util.hooks:Epoch 20\tIter 20\ttrain loss 1.8870840072631836\n"
     ]
    }
   ],
   "source": [
    "fastqa_reader.train(optimizer,\n",
    "                    batch_size=batch_size,\n",
    "                    hooks=hooks,\n",
    "                    max_epochs=20,\n",
    "                    training_set=fastqa_train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions from the trained reader\n",
    "\n",
    "Let's take a look at the predictions after 20 epochs of training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Question: To whom did the Virgin Mary allegedly appear in 1858 in Lourdes France?\n",
      "Answer:   Saint Bernadette Soubirous \t (score: 0.718)\n",
      "\n",
      "Question: What is in front of the Notre Dame Main Building?\n",
      "Answer:   Saint Bernadette Soubirous \t (score: 0.375)\n",
      "\n",
      "Question: The Basilica of the Sacred heart at Notre Dame is beside to which structure?\n",
      "Answer:   a copper statue of Christ \t (score: 0.439)\n",
      "\n",
      "Question: What is the Grotto at Notre Dame?\n",
      "Answer:   Saint Bernadette Soubirous \t (score: 0.367)\n",
      "\n",
      "Question: What sits on top of the Main Building at Notre Dame?\n",
      "Answer:   a copper statue of Christ \t (score: 0.333)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num_questions = 5\n",
    "predictions = fastqa_reader(questions)\n",
    "for q, a in zip(questions[:num_questions], predictions):\n",
    "    print(\"Question: \" + q.question)\n",
    "    print(\"Answer:   %s \\t (score: %.3f)\\n\" % (a[0].text, a[0].score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And let's take a look at one of the answers in the context of the paragraph:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'What entity provides help with the management of time for new students at Notre Dame?'"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "questions[20].question"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "All of Notre Dame's undergraduate students are a part of one of the five undergraduate colleges at the school or are in the First Year of Studies program. The First Year of Studies program was established in 1962 to guide incoming freshmen in their first year at the school before they have declared a major. Each student is given an academic advisor from the program who helps them to choose classes that give them exposure to any major in which they are interested. The program also includes a <span style=\"background-color: #ff00ff; color: white\">Learning Resource Center</span> which provides time management, collaborative learning, and subject tutoring. This program has been recognized previously, by U.S. News & World Report, as outstanding."
      ],
      "text/plain": [
       "<notebooks.prettyprint.QAPrettyPrint at 0x7f950b020cc0>"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "QAPrettyPrint(questions[20].support[0], predictions[20][0].span)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The predicted answers look much better now. However, be aware that this is the prediction of a model trained on a very small subset of data, applied to that same data. Feel free to train your model on the full SQuAD dataset.\n",
    "\n",
    "### Evaluation\n",
    "\n",
    "Let's evaluate our trained model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:jack.core.input_module:OnlineInputModule pre-processes data on-the-fly in first epoch and caches results for subsequent epochs! That means, first epoch might be slower.\n",
      "INFO:jack.core.reader:Start answering...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " [Elapsed Time: 0:00:00] |###################################| (Time: 0:00:00) \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Exact: 0.4166666666666667\n",
      "F1: 0.4351851851851852\n"
     ]
    }
   ],
   "source": [
    "batch_size = len(fastqa_train_data)\n",
    "metrics = evaluate_reader(fastqa_reader, fastqa_train_data, batch_size)\n",
    "pretty_print_results(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving the model\n",
    "\n",
    "We can now save the model after training it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "fastqa_reader.store(\"/tmp/fastqa_reader\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Decomposable attention model (SNLI dataset)\n",
    "\n",
    "We will train a decomposable attention model [[4]](#ref4) on a very small subset of the Stanford NLI dataset [[5]](#ref5).\n",
    "\n",
    "Additionally, please note a simpler way to train the DAM from the command line:\n",
    "\n",
    "> `python3 bin/jack-train.py with config='./conf/nli/snli/dam.yaml'`\n",
    "\n",
    "### Data loading\n",
    "\n",
    "We load the data, and prepare it for later printing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "snli_path = 'data/SNLI/snippet.jtr_v1.json'\n",
    "snli_train_data = load_jack(snli_path)\n",
    "\n",
    "hypotheses = []\n",
    "premises = []\n",
    "labels = []\n",
    "for input_, output_ in snli_train_data:\n",
    "    premises.append(input_.support[0])\n",
    "    hypotheses.append(input_.question)\n",
    "    labels.append(output_[0].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We reset the tensorflow graph to clear out the previously built model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf.reset_default_graph()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating the reader\n",
    "\n",
    "As before, we set up the configuration for the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "snli_config = {\"repr_dim\": 10,\n",
    "               \"reader\": \"dam_snli_reader\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...create the shared resources:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "shared_resources = SharedResources(vocab, snli_config, embeddings)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...build the reader, and set it up with the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:jack.readers.natural_language_inference.decomposable_attention:Building the Attend graph ..\n",
      "INFO:jack.readers.natural_language_inference.decomposable_attention:Building the Compare graph ..\n",
      "INFO:jack.readers.natural_language_inference.decomposable_attention:Building the Aggregate graph ..\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/bos_token_embedding:0' shape=(1, 1, 50) dtype=float32_ref> (Trainable: False)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/transform_embeddings/fully_connected/weights:0' shape=(50, 10) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/attend/transform_attend/fully_connected/weights:0' shape=(10, 10) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/attend/transform_attend/fully_connected/biases:0' shape=(10,) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/attend/transform_attend/fully_connected_1/weights:0' shape=(10, 10) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/attend/transform_attend/fully_connected_1/biases:0' shape=(10,) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/transform_compare/fully_connected/weights:0' shape=(20, 10) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/transform_compare/fully_connected/biases:0' shape=(10,) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/transform_compare/fully_connected_1/weights:0' shape=(10, 10) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/transform_compare/fully_connected_1/biases:0' shape=(10,) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/aggregate/transform_aggregate/fully_connected/weights:0' shape=(20, 10) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/aggregate/transform_aggregate/fully_connected/biases:0' shape=(10,) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/aggregate/transform_aggregate/fully_connected_1/weights:0' shape=(10, 10) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/aggregate/transform_aggregate/fully_connected_1/biases:0' shape=(10,) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/aggregate/fully_connected/weights:0' shape=(10, 3) dtype=float32_ref> (Trainable: True)\n",
      "DEBUG:jack.core.reader:Variable: <tf.Variable 'jtreader/aggregate/fully_connected/biases:0' shape=(3,) dtype=float32_ref> (Trainable: True)\n"
     ]
    }
   ],
   "source": [
    "snli_reader = readers.readers[\"dam_snli_reader\"](shared_resources)\n",
    "snli_reader.setup_from_data(snli_train_data, is_training=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "We set up the training procedure, similarly to the FastQA model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = len(snli_train_data)\n",
    "hooks = [LossHook(snli_reader, iter_interval=1)]\n",
    "optimizer = tf.train.AdamOptimizer(0.02)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "...and run the training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:jack.core.reader:Preparing training data...\n",
      "INFO:jack.core.input_module:OnlineInputModule pre-processes data on-the-fly in first epoch and caches results for subsequent epochs! That means, first epoch might be slower.\n",
      "INFO:jack.core.reader:Number of parameters: 1393\n",
      "INFO:jack.core.reader:Start training...\n",
      "INFO:jack.util.hooks:Epoch 1\tIter 1\ttrain loss 1.0986123085021973\n",
      "INFO:jack.util.hooks:Epoch 2\tIter 2\ttrain loss 1.0960743427276611\n",
      "INFO:jack.util.hooks:Epoch 3\tIter 3\ttrain loss 1.0936973094940186\n",
      "INFO:jack.util.hooks:Epoch 4\tIter 4\ttrain loss 1.0914332866668701\n",
      "INFO:jack.util.hooks:Epoch 5\tIter 5\ttrain loss 1.0889884233474731\n",
      "INFO:jack.util.hooks:Epoch 6\tIter 6\ttrain loss 1.085637092590332\n",
      "INFO:jack.util.hooks:Epoch 7\tIter 7\ttrain loss 1.0806653499603271\n",
      "INFO:jack.util.hooks:Epoch 8\tIter 8\ttrain loss 1.0811249017715454\n",
      "INFO:jack.util.hooks:Epoch 9\tIter 9\ttrain loss 1.0726934671401978\n",
      "INFO:jack.util.hooks:Epoch 10\tIter 10\ttrain loss 1.0655248165130615\n",
      "INFO:jack.util.hooks:Epoch 11\tIter 11\ttrain loss 1.054319143295288\n",
      "INFO:jack.util.hooks:Epoch 12\tIter 12\ttrain loss 1.0329482555389404\n",
      "INFO:jack.util.hooks:Epoch 13\tIter 13\ttrain loss 1.0031580924987793\n",
      "INFO:jack.util.hooks:Epoch 14\tIter 14\ttrain loss 0.9621566534042358\n",
      "INFO:jack.util.hooks:Epoch 15\tIter 15\ttrain loss 0.9214744567871094\n",
      "INFO:jack.util.hooks:Epoch 16\tIter 16\ttrain loss 0.8691341280937195\n",
      "INFO:jack.util.hooks:Epoch 17\tIter 17\ttrain loss 0.7754760980606079\n",
      "INFO:jack.util.hooks:Epoch 18\tIter 18\ttrain loss 0.7177000045776367\n",
      "INFO:jack.util.hooks:Epoch 19\tIter 19\ttrain loss 0.6653531789779663\n",
      "INFO:jack.util.hooks:Epoch 20\tIter 20\ttrain loss 0.6730753183364868\n"
     ]
    }
   ],
   "source": [
    "snli_reader.train(optimizer,\n",
    "                  batch_size=batch_size,\n",
    "                  hooks=hooks,\n",
    "                  max_epochs=20,\n",
    "                  training_set=snli_train_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Predictions from the trained reader"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_ = [qa_setting for qa_setting, answers in snli_train_data]\n",
    "output_ = snli_reader(input_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Premise: A person on a horse jumps over a broken down airplane.\n",
      "Hypothesis: A person is training his horse for a competition.\n",
      "Prediction: neutral (score: 1.00)  [Label: neutral]\n",
      "\n",
      "Premise: A person on a horse jumps over a broken down airplane.\n",
      "Hypothesis: A person is at a diner, ordering an omelette.\n",
      "Prediction: contradiction (score: 0.35)  [Label: contradiction]\n",
      "\n",
      "Premise: A person on a horse jumps over a broken down airplane.\n",
      "Hypothesis: A person is outdoors, on a horse.\n",
      "Prediction: contradiction (score: 0.35)  [Label: entailment]\n",
      "\n",
      "Premise: Children smiling and waving at camera\n",
      "Hypothesis: They are smiling at their parents\n",
      "Prediction: neutral (score: 0.85)  [Label: neutral]\n",
      "\n",
      "Premise: Children smiling and waving at camera\n",
      "Hypothesis: There are children present\n",
      "Prediction: contradiction (score: 0.35)  [Label: entailment]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num_examples = 5\n",
    "for p, h, l, o in zip(premises[:num_examples], hypotheses, labels, output_):\n",
    "    print('Premise: {}'.format(p))\n",
    "    print('Hypothesis: {}'.format(h))\n",
    "    print('Prediction: {} (score: {:.2f})  [Label: {}]\\n'.format(o[0].text, o[0].score, l))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluation\n",
    "\n",
    "Let's evaluate our trained model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:jack.core.input_module:OnlineInputModule pre-processes data on-the-fly in first epoch and caches results for subsequent epochs! That means, first epoch might be slower.\n",
      "INFO:jack.core.reader:Start answering...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      " [Elapsed Time: 0:00:00] |###################################| (Time: 0:00:00) \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.7\n",
      "Confusion Matrix:\n",
      "\n",
      "\t             \tcontradiction\tentailment   \tneutral      \n",
      "\tcontradiction\t3            \t0            \t0            \n",
      "\tentailment   \t3            \t0            \t0            \n",
      "\tneutral      \t0            \t0            \t4            \n",
      "\t\n",
      "F1:\n",
      "\tcontradiction: 0.6666666666666666\n",
      "\tentailment: 0.0\n",
      "\tneutral: 1.0\n",
      "Precision:\n",
      "\tcontradiction: 0.5\n",
      "\tentailment: 0.0\n",
      "\tneutral: 1.0\n",
      "Recall:\n",
      "\tcontradiction: 1.0\n",
      "\tentailment: 0.0\n",
      "\tneutral: 1.0\n"
     ]
    }
   ],
   "source": [
    "batch_size = len(snli_train_data)\n",
    "metrics = evaluate_reader(snli_reader, snli_train_data, batch_size)\n",
    "pretty_print_results(metrics)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References:\n",
    "\n",
    "<a id='ref1'>[1]</a> Jeffrey Pennington, Richard Socher, and Christopher Manning. <a href='http://www.aclweb.org/anthology/D14-1162'>\"Glove: Global vectors for word representation.\"</a> Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). 2014.\n",
    "\n",
    "<a id='ref2'>[2]</a> Dirk Weissenborn, Georg Wiese, and Laura Seiffe. <a href='http://www.aclweb.org/anthology/K17-1028'>\"Making neural qa as simple as possible but not simpler.\"</a> Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL). 2017.</a>\n",
    "\n",
    "<a id='ref3'>[3]</a> Pranav Rajpurkar, et al. <a href='http://www.anthology.aclweb.org/D/D16/D16-1264.pdf'>\"SQuAD: 100,000+ Questions for Machine Comprehension of Text.\"</a> Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2016.\n",
    "\n",
    "<a id='ref4'>[4]</a> Ankur Parikh, Oscar Täckström, Dipanjan Das, Jakob Uszkoreit . <a href='http://www.aclweb.org/anthology/D14-1162'>\"A Decomposable Attention Model for Natural Language Inference.\"</a> Proceedings of the 2016 conference on empirical methods in natural language processing (EMNLP). 2016. \n",
    "\n",
    "<a id='ref5'>[5]</a> Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. <a href='http://www.anthology.aclweb.org/D/D16/D16-1264.pdf'>\"A large annotated corpus for learning natural language inference.\"</a> In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2015."
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}