translation_transformer.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# For tips on running notebooks in Google Colab, see\n",
    "# https://pytorch.org/tutorials/beginner/colab\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Language Translation with ``nn.Transformer`` and torchtext\n",
    "\n",
    "This tutorial shows:\n",
    "    - How to train a translation model from scratch using Transformer.\n",
    "    - Use torchtext library to access  [Multi30k](http://www.statmt.org/wmt16/multimodal-task.html#task1)_ dataset to train a German to English translation model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Sourcing and Processing\n",
    "\n",
    "[torchtext library](https://pytorch.org/text/stable/)_ has utilities for creating datasets that can be easily\n",
    "iterated through for the purposes of creating a language translation\n",
    "model. In this example, we show how to use torchtext's inbuilt datasets,\n",
    "tokenize a raw text sentence, build vocabulary, and numericalize tokens into tensor. We will use\n",
    "[Multi30k dataset from torchtext library](https://pytorch.org/text/stable/datasets.html#multi30k)_\n",
    "that yields a pair of source-target raw sentences.\n",
    "\n",
    "To access torchtext datasets, please install torchdata following instructions at https://github.com/pytorch/data.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torchtext.data.utils import get_tokenizer\n",
    "from torchtext.vocab import build_vocab_from_iterator\n",
    "from torchtext.datasets import multi30k, Multi30k\n",
    "from typing import Iterable, List\n",
    "import spacy\n",
    "\n",
    "\n",
    "# We need to modify the URLs for the dataset since the links to the original dataset are broken\n",
    "# Refer to https://github.com/pytorch/text/issues/1756#issuecomment-1163664163 for more info\n",
    "multi30k.URL[\n",
    "    \"train\"\n",
    "] = \"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/training.tar.gz\"\n",
    "multi30k.URL[\n",
    "    \"valid\"\n",
    "] = \"https://raw.githubusercontent.com/neychev/small_DL_repo/master/datasets/Multi30k/validation.tar.gz\"\n",
    "\n",
    "SRC_LANGUAGE = \"de\"\n",
    "TGT_LANGUAGE = \"en\"\n",
    "\n",
    "# Place-holders\n",
    "token_transform = {}\n",
    "vocab_transform = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%load_ext ersa.client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%run_remote python -m spacy download en_core_web_sm\n",
    "%run_remote python -m spacy download de_core_news_sm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create source and target language tokenizer. Make sure to install the dependencies.\n",
    "\n",
    "```python\n",
    "pip install -U torchdata\n",
    "pip install -U spacy\n",
    "python -m spacy download en_core_web_sm\n",
    "python -m spacy download de_core_news_sm\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "token_transform[SRC_LANGUAGE] = get_tokenizer(\"spacy\", language=\"de_core_news_sm\")\n",
    "token_transform[TGT_LANGUAGE] = get_tokenizer(\"spacy\", language=\"en_core_web_sm\")\n",
    "\n",
    "\n",
    "# helper function to yield list of tokens\n",
    "def yield_tokens(data_iter: Iterable, language: str) -> List[str]:\n",
    "    language_index = {SRC_LANGUAGE: 0, TGT_LANGUAGE: 1}\n",
    "\n",
    "    for data_sample in data_iter:\n",
    "        yield token_transform[language](data_sample[language_index[language]])\n",
    "\n",
    "\n",
    "# Define special symbols and indices\n",
    "UNK_IDX, PAD_IDX, BOS_IDX, EOS_IDX = 0, 1, 2, 3\n",
    "# Make sure the tokens are in order of their indices to properly insert them in vocab\n",
    "special_symbols = [\"<unk>\", \"<pad>\", \"<bos>\", \"<eos>\"]\n",
    "\n",
    "for ln in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
    "    # Training data Iterator\n",
    "    train_iter = Multi30k(split=\"train\", language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
    "    # Create torchtext's Vocab object\n",
    "    vocab_transform[ln] = build_vocab_from_iterator(\n",
    "        yield_tokens(train_iter, ln),\n",
    "        min_freq=1,\n",
    "        specials=special_symbols,\n",
    "        special_first=True,\n",
    "    )\n",
    "\n",
    "# Set ``UNK_IDX`` as the default index. This index is returned when the token is not found.\n",
    "# If not set, it throws ``RuntimeError`` when the queried token is not found in the Vocabulary.\n",
    "for ln in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
    "    vocab_transform[ln].set_default_index(UNK_IDX)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Seq2Seq Network using Transformer\n",
    "\n",
    "Transformer is a Seq2Seq model introduced in [“Attention is all you\n",
    "need”](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)_\n",
    "paper for solving machine translation tasks.\n",
    "Below, we will create a Seq2Seq network that uses Transformer. The network\n",
    "consists of three parts. First part is the embedding layer. This layer converts tensor of input indices\n",
    "into corresponding tensor of input embeddings. These embedding are further augmented with positional\n",
    "encodings to provide position information of input tokens to the model. The second part is the\n",
    "actual [Transformer](https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html)_ model.\n",
    "Finally, the output of the Transformer model is passed through linear layer\n",
    "that gives unnormalized probabilities for each token in the target language.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch import Tensor\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "from torch.nn import Transformer\n",
    "import math\n",
    "\n",
    "DEVICE = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "\n",
    "\n",
    "# helper Module that adds positional encoding to the token embedding to introduce a notion of word order.\n",
    "class PositionalEncoding(nn.Module):\n",
    "    def __init__(self, emb_size: int, dropout: float, maxlen: int = 5000):\n",
    "        super(PositionalEncoding, self).__init__()\n",
    "        den = torch.exp(-torch.arange(0, emb_size, 2) * math.log(10000) / emb_size)\n",
    "        pos = torch.arange(0, maxlen).reshape(maxlen, 1)\n",
    "        pos_embedding = torch.zeros((maxlen, emb_size))\n",
    "        pos_embedding[:, 0::2] = torch.sin(pos * den)\n",
    "        pos_embedding[:, 1::2] = torch.cos(pos * den)\n",
    "        pos_embedding = pos_embedding.unsqueeze(-2)\n",
    "\n",
    "        self.dropout = nn.Dropout(dropout)\n",
    "        self.register_buffer(\"pos_embedding\", pos_embedding)\n",
    "\n",
    "    def forward(self, token_embedding: Tensor):\n",
    "        return self.dropout(\n",
    "            token_embedding + self.pos_embedding[: token_embedding.size(0), :]\n",
    "        )\n",
    "\n",
    "\n",
    "# helper Module to convert tensor of input indices into corresponding tensor of token embeddings\n",
    "class TokenEmbedding(nn.Module):\n",
    "    def __init__(self, vocab_size: int, emb_size):\n",
    "        super(TokenEmbedding, self).__init__()\n",
    "        self.embedding = nn.Embedding(vocab_size, emb_size)\n",
    "        self.emb_size = emb_size\n",
    "\n",
    "    def forward(self, tokens: Tensor):\n",
    "        return self.embedding(tokens.long()) * math.sqrt(self.emb_size)\n",
    "\n",
    "\n",
    "# Seq2Seq Network\n",
    "class Seq2SeqTransformer(nn.Module):\n",
    "    def __init__(\n",
    "        self,\n",
    "        num_encoder_layers: int,\n",
    "        num_decoder_layers: int,\n",
    "        emb_size: int,\n",
    "        nhead: int,\n",
    "        src_vocab_size: int,\n",
    "        tgt_vocab_size: int,\n",
    "        dim_feedforward: int = 512,\n",
    "        dropout: float = 0.1,\n",
    "    ):\n",
    "        super(Seq2SeqTransformer, self).__init__()\n",
    "        self.transformer = Transformer(\n",
    "            d_model=emb_size,\n",
    "            nhead=nhead,\n",
    "            num_encoder_layers=num_encoder_layers,\n",
    "            num_decoder_layers=num_decoder_layers,\n",
    "            dim_feedforward=dim_feedforward,\n",
    "            dropout=dropout,\n",
    "        )\n",
    "        self.generator = nn.Linear(emb_size, tgt_vocab_size)\n",
    "        self.src_tok_emb = TokenEmbedding(src_vocab_size, emb_size)\n",
    "        self.tgt_tok_emb = TokenEmbedding(tgt_vocab_size, emb_size)\n",
    "        self.positional_encoding = PositionalEncoding(emb_size, dropout=dropout)\n",
    "\n",
    "    def forward(\n",
    "        self,\n",
    "        src: Tensor,\n",
    "        trg: Tensor,\n",
    "        src_mask: Tensor,\n",
    "        tgt_mask: Tensor,\n",
    "        src_padding_mask: Tensor,\n",
    "        tgt_padding_mask: Tensor,\n",
    "        memory_key_padding_mask: Tensor,\n",
    "    ):\n",
    "        src_emb = self.positional_encoding(self.src_tok_emb(src))\n",
    "        tgt_emb = self.positional_encoding(self.tgt_tok_emb(trg))\n",
    "        outs = self.transformer(\n",
    "            src_emb,\n",
    "            tgt_emb,\n",
    "            src_mask,\n",
    "            tgt_mask,\n",
    "            None,\n",
    "            src_padding_mask,\n",
    "            tgt_padding_mask,\n",
    "            memory_key_padding_mask,\n",
    "        )\n",
    "        return self.generator(outs)\n",
    "\n",
    "    def encode(self, src: Tensor, src_mask: Tensor):\n",
    "        return self.transformer.encoder(\n",
    "            self.positional_encoding(self.src_tok_emb(src)), src_mask\n",
    "        )\n",
    "\n",
    "    def decode(self, tgt: Tensor, memory: Tensor, tgt_mask: Tensor):\n",
    "        return self.transformer.decoder(\n",
    "            self.positional_encoding(self.tgt_tok_emb(tgt)), memory, tgt_mask\n",
    "        )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "During training, we need a subsequent word mask that will prevent the model from looking into\n",
    "the future words when making predictions. We will also need masks to hide\n",
    "source and target padding tokens. Below, let's define a function that will take care of both.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_square_subsequent_mask(sz):\n",
    "    mask = (torch.triu(torch.ones((sz, sz), device=DEVICE)) == 1).transpose(0, 1)\n",
    "    mask = (\n",
    "        mask.float()\n",
    "        .masked_fill(mask == 0, float(\"-inf\"))\n",
    "        .masked_fill(mask == 1, float(0.0))\n",
    "    )\n",
    "    return mask\n",
    "\n",
    "\n",
    "def create_mask(src, tgt):\n",
    "    src_seq_len = src.shape[0]\n",
    "    tgt_seq_len = tgt.shape[0]\n",
    "\n",
    "    tgt_mask = generate_square_subsequent_mask(tgt_seq_len)\n",
    "    src_mask = torch.zeros((src_seq_len, src_seq_len), device=DEVICE).type(torch.bool)\n",
    "\n",
    "    src_padding_mask = (src == PAD_IDX).transpose(0, 1)\n",
    "    tgt_padding_mask = (tgt == PAD_IDX).transpose(0, 1)\n",
    "    return src_mask, tgt_mask, src_padding_mask, tgt_padding_mask"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now define the parameters of our model and instantiate the same. Below, we also\n",
    "define our loss function which is the cross-entropy loss and the optimizer used for training.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.manual_seed(0)\n",
    "\n",
    "SRC_VOCAB_SIZE = len(vocab_transform[SRC_LANGUAGE])\n",
    "TGT_VOCAB_SIZE = len(vocab_transform[TGT_LANGUAGE])\n",
    "EMB_SIZE = 512\n",
    "NHEAD = 8\n",
    "FFN_HID_DIM = 512\n",
    "BATCH_SIZE = 128\n",
    "NUM_ENCODER_LAYERS = 3\n",
    "NUM_DECODER_LAYERS = 3\n",
    "\n",
    "transformer = Seq2SeqTransformer(\n",
    "    NUM_ENCODER_LAYERS,\n",
    "    NUM_DECODER_LAYERS,\n",
    "    EMB_SIZE,\n",
    "    NHEAD,\n",
    "    SRC_VOCAB_SIZE,\n",
    "    TGT_VOCAB_SIZE,\n",
    "    FFN_HID_DIM,\n",
    ")\n",
    "\n",
    "for p in transformer.parameters():\n",
    "    if p.dim() > 1:\n",
    "        nn.init.xavier_uniform_(p)\n",
    "\n",
    "transformer = transformer.to(DEVICE)\n",
    "\n",
    "loss_fn = torch.nn.CrossEntropyLoss(ignore_index=PAD_IDX)\n",
    "\n",
    "optimizer = torch.optim.Adam(\n",
    "    transformer.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collation\n",
    "\n",
    "As seen in the ``Data Sourcing and Processing`` section, our data iterator yields a pair of raw strings.\n",
    "We need to convert these string pairs into the batched tensors that can be processed by our ``Seq2Seq`` network\n",
    "defined previously. Below we define our collate function that converts a batch of raw strings into batch tensors that\n",
    "can be fed directly into our model.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.nn.utils.rnn import pad_sequence\n",
    "\n",
    "\n",
    "# helper function to club together sequential operations\n",
    "def sequential_transforms(*transforms):\n",
    "    def func(txt_input):\n",
    "        for transform in transforms:\n",
    "            txt_input = transform(txt_input)\n",
    "        return txt_input\n",
    "\n",
    "    return func\n",
    "\n",
    "\n",
    "# function to add BOS/EOS and create tensor for input sequence indices\n",
    "def tensor_transform(token_ids: List[int]):\n",
    "    return torch.cat(\n",
    "        (torch.tensor([BOS_IDX]), torch.tensor(token_ids), torch.tensor([EOS_IDX]))\n",
    "    )\n",
    "\n",
    "\n",
    "# ``src`` and ``tgt`` language text transforms to convert raw strings into tensors indices\n",
    "text_transform = {}\n",
    "for ln in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
    "    text_transform[ln] = sequential_transforms(\n",
    "        token_transform[ln],  # Tokenization\n",
    "        vocab_transform[ln],  # Numericalization\n",
    "        tensor_transform,\n",
    "    )  # Add BOS/EOS and create tensor\n",
    "\n",
    "\n",
    "# function to collate data samples into batch tensors\n",
    "def collate_fn(batch):\n",
    "    src_batch, tgt_batch = [], []\n",
    "    for src_sample, tgt_sample in batch:\n",
    "        src_batch.append(text_transform[SRC_LANGUAGE](src_sample.rstrip(\"\\n\")))\n",
    "        tgt_batch.append(text_transform[TGT_LANGUAGE](tgt_sample.rstrip(\"\\n\")))\n",
    "\n",
    "    src_batch = pad_sequence(src_batch, padding_value=PAD_IDX)\n",
    "    tgt_batch = pad_sequence(tgt_batch, padding_value=PAD_IDX)\n",
    "    return src_batch, tgt_batch"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's define training and evaluation loop that will be called for each\n",
    "epoch.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.utils.data import DataLoader\n",
    "\n",
    "\n",
    "def train_epoch(model, optimizer):\n",
    "    model.train()\n",
    "    losses = 0\n",
    "    train_iter = Multi30k(split=\"train\", language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
    "    train_dataloader = DataLoader(\n",
    "        train_iter, batch_size=BATCH_SIZE, collate_fn=collate_fn\n",
    "    )\n",
    "\n",
    "    for src, tgt in train_dataloader:\n",
    "        src = src.to(DEVICE)\n",
    "        tgt = tgt.to(DEVICE)\n",
    "\n",
    "        tgt_input = tgt[:-1, :]\n",
    "\n",
    "        src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(\n",
    "            src, tgt_input\n",
    "        )\n",
    "\n",
    "        logits = model(\n",
    "            src,\n",
    "            tgt_input,\n",
    "            src_mask,\n",
    "            tgt_mask,\n",
    "            src_padding_mask,\n",
    "            tgt_padding_mask,\n",
    "            src_padding_mask,\n",
    "        )\n",
    "\n",
    "        optimizer.zero_grad()\n",
    "\n",
    "        tgt_out = tgt[1:, :]\n",
    "        loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))\n",
    "        loss.backward()\n",
    "\n",
    "        optimizer.step()\n",
    "        losses += loss.item()\n",
    "\n",
    "    return losses / len(list(train_dataloader))\n",
    "\n",
    "\n",
    "def evaluate(model):\n",
    "    model.eval()\n",
    "    losses = 0\n",
    "\n",
    "    val_iter = Multi30k(split=\"valid\", language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
    "    val_dataloader = DataLoader(val_iter, batch_size=BATCH_SIZE, collate_fn=collate_fn)\n",
    "\n",
    "    for src, tgt in val_dataloader:\n",
    "        src = src.to(DEVICE)\n",
    "        tgt = tgt.to(DEVICE)\n",
    "\n",
    "        tgt_input = tgt[:-1, :]\n",
    "\n",
    "        src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(\n",
    "            src, tgt_input\n",
    "        )\n",
    "\n",
    "        logits = model(\n",
    "            src,\n",
    "            tgt_input,\n",
    "            src_mask,\n",
    "            tgt_mask,\n",
    "            src_padding_mask,\n",
    "            tgt_padding_mask,\n",
    "            src_padding_mask,\n",
    "        )\n",
    "\n",
    "        tgt_out = tgt[1:, :]\n",
    "        loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))\n",
    "        losses += loss.item()\n",
    "\n",
    "    return losses / len(list(val_dataloader))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have all the ingredients to train our model. Let's do it!\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from timeit import default_timer as timer\n",
    "\n",
    "NUM_EPOCHS = 18\n",
    "\n",
    "for epoch in range(1, NUM_EPOCHS + 1):\n",
    "    start_time = timer()\n",
    "    train_loss = train_epoch(transformer, optimizer)\n",
    "    end_time = timer()\n",
    "    val_loss = evaluate(transformer)\n",
    "    print(\n",
    "        (\n",
    "            f\"Epoch: {epoch}, Train loss: {train_loss:.3f}, Val loss: {val_loss:.3f}, \"\n",
    "            f\"Epoch time = {(end_time - start_time):.3f}s\"\n",
    "        )\n",
    "    )\n",
    "\n",
    "\n",
    "# function to generate output sequence using greedy algorithm\n",
    "def greedy_decode(model, src, src_mask, max_len, start_symbol):\n",
    "    src = src.to(DEVICE)\n",
    "    src_mask = src_mask.to(DEVICE)\n",
    "\n",
    "    memory = model.encode(src, src_mask)\n",
    "    ys = torch.ones(1, 1).fill_(start_symbol).type(torch.long).to(DEVICE)\n",
    "    for i in range(max_len - 1):\n",
    "        memory = memory.to(DEVICE)\n",
    "        tgt_mask = (generate_square_subsequent_mask(ys.size(0)).type(torch.bool)).to(\n",
    "            DEVICE\n",
    "        )\n",
    "        out = model.decode(ys, memory, tgt_mask)\n",
    "        out = out.transpose(0, 1)\n",
    "        prob = model.generator(out[:, -1])\n",
    "        _, next_word = torch.max(prob, dim=1)\n",
    "        next_word = next_word.item()\n",
    "\n",
    "        ys = torch.cat([ys, torch.ones(1, 1).type_as(src.data).fill_(next_word)], dim=0)\n",
    "        if next_word == EOS_IDX:\n",
    "            break\n",
    "    return ys\n",
    "\n",
    "\n",
    "# actual function to translate input sentence into target language\n",
    "def translate(model: torch.nn.Module, src_sentence: str):\n",
    "    model.eval()\n",
    "    src = text_transform[SRC_LANGUAGE](src_sentence).view(-1, 1)\n",
    "    num_tokens = src.shape[0]\n",
    "    src_mask = (torch.zeros(num_tokens, num_tokens)).type(torch.bool)\n",
    "    tgt_tokens = greedy_decode(\n",
    "        model, src, src_mask, max_len=num_tokens + 5, start_symbol=BOS_IDX\n",
    "    ).flatten()\n",
    "    return (\n",
    "        \" \".join(\n",
    "            vocab_transform[TGT_LANGUAGE].lookup_tokens(list(tgt_tokens.cpu().numpy()))\n",
    "        )\n",
    "        .replace(\"<bos>\", \"\")\n",
    "        .replace(\"<eos>\", \"\")\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(translate(transformer, \"Eine Gruppe von Menschen steht vor einem Iglu .\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References\n",
    "\n",
    "1. Attention is all you need paper.\n",
    "   https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf\n",
    "2. The annotated transformer. https://nlp.seas.harvard.edu/2018/04/03/attention.html#positional-encoding\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}