Copy_of_Tacotron_Training_Notebook.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "4kK32tPFFATp"
      },
      "source": [
        "# Tacotron Training Notebook"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tK3xiflQHZqz"
      },
      "source": [
        "First, go up to \"Files\", and click \"Save copy to Drive\", and go to the copy, then continue."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "RwSHGNXWEjys"
      },
      "source": [
        "# Check GPU\n",
        "\n",
        "First, if you want, you can check to see which GPU is currently being used. The best GPUs are P100, V100, and T4. If you get a K80 or P4, you can restart the runtime and try again if you'd like, but all GPUs will (probably) work with this notebook."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1lP4tdlKEEqU",
        "outputId": "98ae5ef9-8910-448d-e4a8-0fbfa8301408",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "GPU 0: Tesla T4 (UUID: GPU-baf1059c-0bbf-f080-ed38-9a026984bc27)\n"
          ]
        }
      ],
      "source": [
        "!nvidia-smi -L"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "sZxLjK10FrMy"
      },
      "source": [
        "#Mount Your Google Drive\n",
        "\n",
        "You will need to mount your Google Drive for this notebook. This is where your models are saved. If you're planning on making a lot of Tacotron models, I would reccommend you get more Google Drive storage. Each model is 300 MB.\n",
        "\n",
        "This cell will put a link to the Google Drive website, and it will ask you if you want to link your Google Drive. Afterwards, it will give you a code. Copy and paste it into the text box below the output."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 83,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "lbZw-LNjF3f0",
        "outputId": "432a0760-7869-402e-a559-4f3e7f6ec7e8",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Mounted at drive\n"
          ]
        }
      ],
      "source": [
        "from google.colab import drive\n",
        "drive.mount('drive', force_remount=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "SjtxzCV2Gywb"
      },
      "source": [
        "# Donwload Tacotron"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vTVe3hL2G4VE",
        "outputId": "4da248f2-1c70-473e-c332-562eaeafd952",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Submodule 'waveglow' (https://github.com/NVIDIA/waveglow) registered for path 'waveglow'\n",
            "Cloning into '/content/tacotron2/waveglow'...\n",
            "Submodule path 'waveglow': checked out '5bc2a53e20b3b533362f974cfa1ea0267ae1c2b1'\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m235.9/235.9 kB\u001b[0m \u001b[31m18.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m114.5/114.5 kB\u001b[0m \u001b[31m17.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n",
            "\u001b[?25h"
          ]
        }
      ],
      "source": [
        "import os\n",
        "!git clone -q https://github.com/NVIDIA/tacotron2\n",
        "os.chdir('tacotron2')\n",
        "!git submodule init\n",
        "!git submodule update\n",
        "!pip install -q unidecode tensorboardX"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "nk5eQe8BHugf"
      },
      "source": [
        "# Upload dataset\n",
        "\n",
        "This is where you will need to upload your dataset. First, run the cell below. This will make the folder where you will put your audio files."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KvZRk3v4H324",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "data_path = 'wavs'\n",
        "!mkdir {data_path}"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "UQc_8ISExbw-",
        "outputId": "4c69ffeb-62f7-4e81-da97-33397a372f44",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "--2023-04-12 11:09:31--  http://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2\n",
            "Resolving data.keithito.com (data.keithito.com)... 174.138.79.61\n",
            "Connecting to data.keithito.com (data.keithito.com)|174.138.79.61|:80... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 2748572632 (2.6G) [application/octet-stream]\n",
            "Saving to: ‘/content/tacotron2/wavs/LJSpeech-1.1.tar.bz2’\n",
            "\n",
            "LJSpeech-1.1.tar.bz 100%[===================>]   2.56G  13.2MB/s    in 3m 20s  \n",
            "\n",
            "2023-04-12 11:12:52 (13.1 MB/s) - ‘/content/tacotron2/wavs/LJSpeech-1.1.tar.bz2’ saved [2748572632/2748572632]\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!wget -P /content/tacotron2/wavs http://data.keithito.com/data/speech/LJSpeech-1.1.tar.bz2"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CIOe_88IK6iF"
      },
      "source": [
        "# Prepare the model\n",
        "\n",
        "This next section prepares the model and stuff for training.\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "QPKiD1QFAsKy",
        "outputId": "11c7458d-cecb-41de-a97b-fd49d0267541",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "2023-04-12 11:20:58.771697: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.\n",
            "To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.\n",
            "2023-04-12 11:20:59.767481: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT\n",
            "WARNING line 115:4: *.save requires manual check. (This warning is only applicable if the code saves a tf.Keras model) Keras model.save now saves to the Tensorflow SavedModel format by default, instead of HDF5. To continue saving to HDF5, add the argument save_format='h5' to the save() function.\n",
            "WARNING line 71:4: *.save requires manual check. (This warning is only applicable if the code saves a tf.Keras model) Keras model.save now saves to the Tensorflow SavedModel format by default, instead of HDF5. To continue saving to HDF5, add the argument save_format='h5' to the save() function.\n",
            "WARNING line 142:8: *.save requires manual check. (This warning is only applicable if the code saves a tf.Keras model) Keras model.save now saves to the Tensorflow SavedModel format by default, instead of HDF5. To continue saving to HDF5, add the argument save_format='h5' to the save() function.\n",
            "WARNING line 57:4: *.save requires manual check. (This warning is only applicable if the code saves a tf.Keras model) Keras model.save now saves to the Tensorflow SavedModel format by default, instead of HDF5. To continue saving to HDF5, add the argument save_format='h5' to the save() function.\n",
            "TensorFlow 2.0 Upgrade Script\n",
            "-----------------------------\n",
            "Converted 27 files\n",
            "Detected 4 issues that require attention\n",
            "--------------------------------------------------------------------------------\n",
            "--------------------------------------------------------------------------------\n",
            "File: /content/tacotron2/train.py\n",
            "--------------------------------------------------------------------------------\n",
            "/content/tacotron2/train.py:115:4: WARNING: *.save requires manual check. (This warning is only applicable if the code saves a tf.Keras model) Keras model.save now saves to the Tensorflow SavedModel format by default, instead of HDF5. To continue saving to HDF5, add the argument save_format='h5' to the save() function.\n",
            "--------------------------------------------------------------------------------\n",
            "File: /content/tacotron2/waveglow/convert_model.py\n",
            "--------------------------------------------------------------------------------\n",
            "/content/tacotron2/waveglow/convert_model.py:71:4: WARNING: *.save requires manual check. (This warning is only applicable if the code saves a tf.Keras model) Keras model.save now saves to the Tensorflow SavedModel format by default, instead of HDF5. To continue saving to HDF5, add the argument save_format='h5' to the save() function.\n",
            "--------------------------------------------------------------------------------\n",
            "File: /content/tacotron2/waveglow/mel2samp.py\n",
            "--------------------------------------------------------------------------------\n",
            "/content/tacotron2/waveglow/mel2samp.py:142:8: WARNING: *.save requires manual check. (This warning is only applicable if the code saves a tf.Keras model) Keras model.save now saves to the Tensorflow SavedModel format by default, instead of HDF5. To continue saving to HDF5, add the argument save_format='h5' to the save() function.\n",
            "--------------------------------------------------------------------------------\n",
            "File: /content/tacotron2/waveglow/train.py\n",
            "--------------------------------------------------------------------------------\n",
            "/content/tacotron2/waveglow/train.py:57:4: WARNING: *.save requires manual check. (This warning is only applicable if the code saves a tf.Keras model) Keras model.save now saves to the Tensorflow SavedModel format by default, instead of HDF5. To continue saving to HDF5, add the argument save_format='h5' to the save() function.\n",
            "\n",
            "\n",
            "Make sure to read the detailed log 'report.txt'\n",
            "\n"
          ]
        }
      ],
      "source": [
        "!tf_upgrade_v2 \\\n",
        "--intree /content/tacotron2/ \\\n",
        "--outtree /content/tacotron2/ \\\n",
        "--reportfile report.txt"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 3,
      "metadata": {
        "cellView": "form",
        "id": "cD7wNMRZKrJ3",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "#@title A bunch of boring code and stuff (if you want to look at it, double click on this cell)\n",
        "%matplotlib inline\n",
        "import os\n",
        "if os.getcwd() != '/content/tacotron2':\n",
        "    os.chdir('tacotron2')\n",
        "import time\n",
        "import argparse\n",
        "import math\n",
        "from numpy import finfo\n",
        "\n",
        "import torch\n",
        "from distributed import apply_gradient_allreduce\n",
        "import torch.distributed as dist\n",
        "from torch.utils.data.distributed import DistributedSampler\n",
        "from torch.utils.data import DataLoader\n",
        "\n",
        "from model import Tacotron2\n",
        "from data_utils import TextMelLoader, TextMelCollate\n",
        "from loss_function import Tacotron2Loss\n",
        "from logger import Tacotron2Logger\n",
        "from hparams import create_hparams\n",
        " \n",
        "import random\n",
        "import numpy as np\n",
        "\n",
        "import layers\n",
        "from utils import load_wav_to_torch, load_filepaths_and_text\n",
        "from text import text_to_sequence\n",
        "from math import e\n",
        "#from tqdm import tqdm # Terminal\n",
        "#from tqdm import tqdm_notebook as tqdm # Legacy Notebook TQDM\n",
        "from tqdm.notebook import tqdm # Modern Notebook TQDM\n",
        "from distutils.dir_util import copy_tree\n",
        "import matplotlib.pylab as plt\n",
        "import requests\n",
        "import subprocess\n",
        "def download_from_google_drive(file_id, file_name):\n",
        "  # download a file from the Google Drive link\n",
        "  !rm -f ./cookie\n",
        "  !curl -c ./cookie -s -L \"https://drive.google.com/uc?export=download&id=1c5ZTuT7J08wLUoVZ2KkUs_VdZuJ86ZqA\" > /dev/null\n",
        "  confirm_text = !awk '/download/ {print $NF}' ./cookie\n",
        "  confirm_text = confirm_text[0]\n",
        "  !curl -Lb ./cookie \"https://drive.google.com/uc?export=download&confirm={confirm_text}&id={file_id}\" -o {file_name}\n",
        "\n",
        "\n",
        "def create_mels():\n",
        "    print(\"Generating Mels\")\n",
        "    stft = layers.TacotronSTFT(\n",
        "                hparams.filter_length, hparams.hop_length, hparams.win_length,\n",
        "                hparams.n_mel_channels, hparams.sampling_rate, hparams.mel_fmin,\n",
        "                hparams.mel_fmax)\n",
        "    def save_mel(filename):\n",
        "        audio, sampling_rate = load_wav_to_torch(filename)\n",
        "        if sampling_rate != stft.sampling_rate:\n",
        "            raise ValueError(\"{} {} SR doesn't match target {} SR\".format(filename, \n",
        "                sampling_rate, stft.sampling_rate))\n",
        "        audio_norm = audio / hparams.max_wav_value\n",
        "        audio_norm = audio_norm.unsqueeze(0)\n",
        "        audio_norm = torch.autograd.Variable(audio_norm, requires_grad=False)\n",
        "        melspec = stft.mel_spectrogram(audio_norm)\n",
        "        melspec = torch.squeeze(melspec, 0).cpu().numpy()\n",
        "        np.save(filename.replace('.wav', ''), melspec)\n",
        "\n",
        "    import glob\n",
        "    wavs = glob.glob('wavs/*.wav')\n",
        "    for i in tqdm(wavs):\n",
        "        save_mel(i)\n",
        "\n",
        "\n",
        "def reduce_tensor(tensor, n_gpus):\n",
        "    rt = tensor.clone()\n",
        "    dist.all_reduce(rt, op=dist.reduce_op.SUM)\n",
        "    rt /= n_gpus\n",
        "    return rt\n",
        "\n",
        "\n",
        "def init_distributed(hparams, n_gpus, rank, group_name):\n",
        "    assert torch.cuda.is_available(), \"Distributed mode requires CUDA.\"\n",
        "    print(\"Initializing Distributed\")\n",
        "\n",
        "    # Set cuda device so everything is done on the right GPU.\n",
        "    torch.cuda.set_device(rank % torch.cuda.device_count())\n",
        "\n",
        "    # Initialize distributed communication\n",
        "    dist.init_process_group(\n",
        "        backend=hparams.dist_backend, init_method=hparams.dist_url,\n",
        "        world_size=n_gpus, rank=rank, group_name=group_name)\n",
        "\n",
        "    print(\"Done initializing distributed\")\n",
        "\n",
        "\n",
        "def prepare_dataloaders(hparams):\n",
        "    # Get data, data loaders and collate function ready\n",
        "    trainset = TextMelLoader(hparams.training_files, hparams)\n",
        "    valset = TextMelLoader(hparams.validation_files, hparams)\n",
        "    collate_fn = TextMelCollate(hparams.n_frames_per_step)\n",
        "\n",
        "    if hparams.distributed_run:\n",
        "        train_sampler = DistributedSampler(trainset)\n",
        "        shuffle = False\n",
        "    else:\n",
        "        train_sampler = None\n",
        "        shuffle = True\n",
        "\n",
        "    train_loader = DataLoader(trainset, num_workers=1, shuffle=shuffle,\n",
        "                              sampler=train_sampler,\n",
        "                              batch_size=hparams.batch_size, pin_memory=False,\n",
        "                              drop_last=True, collate_fn=collate_fn)\n",
        "    return train_loader, valset, collate_fn\n",
        "\n",
        "\n",
        "def prepare_directories_and_logger(output_directory, log_directory, rank):\n",
        "    if rank == 0:\n",
        "        if not os.path.isdir(output_directory):\n",
        "            os.makedirs(output_directory)\n",
        "            os.chmod(output_directory, 0o775)\n",
        "        logger = Tacotron2Logger(os.path.join(output_directory, log_directory))\n",
        "    else:\n",
        "        logger = None\n",
        "    return logger\n",
        "\n",
        "\n",
        "def load_model(hparams):\n",
        "    model = Tacotron2(hparams).cuda()\n",
        "    if hparams.fp16_run:\n",
        "        model.decoder.attention_layer.score_mask_value = finfo('float16').min\n",
        "\n",
        "    if hparams.distributed_run:\n",
        "        model = apply_gradient_allreduce(model)\n",
        "\n",
        "    return model\n",
        "\n",
        "\n",
        "def warm_start_model(checkpoint_path, model, ignore_layers):\n",
        "    assert os.path.isfile(checkpoint_path)\n",
        "    print(\"Warm starting model from checkpoint '{}'\".format(checkpoint_path))\n",
        "    checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')\n",
        "    model_dict = checkpoint_dict['state_dict']\n",
        "    if len(ignore_layers) > 0:\n",
        "        model_dict = {k: v for k, v in model_dict.items()\n",
        "                      if k not in ignore_layers}\n",
        "        dummy_dict = model.state_dict()\n",
        "        dummy_dict.update(model_dict)\n",
        "        model_dict = dummy_dict\n",
        "    model.load_state_dict(model_dict)\n",
        "    return model\n",
        "\n",
        "\n",
        "def load_checkpoint(checkpoint_path, model, optimizer):\n",
        "    # assert os.path.isfile(checkpoint_path)\n",
        "    print(\"Loading checkpoint '{}'\".format(checkpoint_path))\n",
        "    checkpoint_dict = torch.load(checkpoint_path, map_location='cpu')\n",
        "    model.load_state_dict(checkpoint_dict['state_dict'])\n",
        "    optimizer.load_state_dict(checkpoint_dict['optimizer'])\n",
        "    learning_rate = checkpoint_dict['learning_rate']\n",
        "    iteration = checkpoint_dict['iteration']\n",
        "    print(\"Loaded checkpoint '{}' from iteration {}\" .format(\n",
        "        checkpoint_path, iteration))\n",
        "    return model, optimizer, learning_rate, iteration\n",
        "\n",
        "\n",
        "def save_checkpoint(model, optimizer, learning_rate, iteration, filepath):\n",
        "    print(\"Saving model and optimizer state at iteration {} to {}\".format(\n",
        "        iteration, filepath))\n",
        "    try:\n",
        "        torch.save({'iteration': iteration,\n",
        "                'state_dict': model.state_dict(),\n",
        "                'optimizer': optimizer.state_dict(),\n",
        "                'learning_rate': learning_rate}, filepath)\n",
        "    except KeyboardInterrupt:\n",
        "        print(\"interrupt received while saving, waiting for save to complete.\")\n",
        "        torch.save({'iteration': iteration,'state_dict': model.state_dict(),'optimizer': optimizer.state_dict(),'learning_rate': learning_rate}, filepath)\n",
        "    print(\"Model Saved\")\n",
        "\n",
        "def plot_alignment(alignment, info=None):\n",
        "    %matplotlib inline\n",
        "    fig, ax = plt.subplots(figsize=(int(alignment_graph_width/100), int(alignment_graph_height/100)))\n",
        "    im = ax.imshow(alignment, cmap='inferno', aspect='auto', origin='lower',\n",
        "                   interpolation='none')\n",
        "    ax.autoscale(enable=True, axis=\"y\", tight=True)\n",
        "    fig.colorbar(im, ax=ax)\n",
        "    xlabel = 'Decoder timestep'\n",
        "    if info is not None:\n",
        "        xlabel += '\\n\\n' + info\n",
        "    plt.xlabel(xlabel)\n",
        "    plt.ylabel('Encoder timestep')\n",
        "    plt.tight_layout()\n",
        "    fig.canvas.draw()\n",
        "    plt.show()\n",
        "\n",
        "def validate(model, criterion, valset, iteration, batch_size, n_gpus,\n",
        "             collate_fn, logger, distributed_run, rank, epoch, start_eposh, learning_rate):\n",
        "    \"\"\"Handles all the validation scoring and printing\"\"\"\n",
        "    model.eval()\n",
        "    with torch.no_grad():\n",
        "        val_sampler = DistributedSampler(valset) if distributed_run else None\n",
        "        val_loader = DataLoader(valset, sampler=val_sampler, num_workers=1,\n",
        "                                shuffle=False, batch_size=batch_size,\n",
        "                                pin_memory=False, collate_fn=collate_fn)\n",
        "\n",
        "        val_loss = 0.0\n",
        "        for i, batch in enumerate(val_loader):\n",
        "            x, y = model.parse_batch(batch)\n",
        "            y_pred = model(x)\n",
        "            loss = criterion(y_pred, y)\n",
        "            if distributed_run:\n",
        "                reduced_val_loss = reduce_tensor(loss.data, n_gpus).item()\n",
        "            else:\n",
        "                reduced_val_loss = loss.item()\n",
        "            val_loss += reduced_val_loss\n",
        "        val_loss = val_loss / (i + 1)\n",
        "\n",
        "    model.train()\n",
        "    if rank == 0:\n",
        "        print(\"Epoch: {} Validation loss {}: {:9f}  Time: {:.1f}m LR: {:.6f}\".format(epoch, iteration, val_loss,(time.perf_counter()-start_eposh)/60, learning_rate))\n",
        "        logger.log_validation(val_loss, model, y, y_pred, iteration)\n",
        "        if hparams.show_alignments:\n",
        "            %matplotlib inline\n",
        "            _, mel_outputs, gate_outputs, alignments = y_pred\n",
        "            idx = random.randint(0, alignments.size(0) - 1)\n",
        "            plot_alignment(alignments[idx].data.cpu().numpy().T)\n",
        "\n",
        "def train(output_directory, log_directory, checkpoint_path, warm_start, n_gpus,\n",
        "          rank, group_name, hparams, log_directory2):\n",
        "    \"\"\"Training and validation logging results to tensorboard and stdout\n",
        "\n",
        "    Params\n",
        "    ------\n",
        "    output_directory (string): directory to save checkpoints\n",
        "    log_directory (string) directory to save tensorboard logs\n",
        "    checkpoint_path(string): checkpoint path\n",
        "    n_gpus (int): number of gpus\n",
        "    rank (int): rank of current gpu\n",
        "    hparams (object): comma separated list of \"name=value\" pairs.\n",
        "    \"\"\"\n",
        "    if hparams.distributed_run:\n",
        "        init_distributed(hparams, n_gpus, rank, group_name)\n",
        "\n",
        "    torch.manual_seed(hparams.seed)\n",
        "    torch.cuda.manual_seed(hparams.seed)\n",
        "\n",
        "    model = load_model(hparams)\n",
        "    learning_rate = hparams.learning_rate\n",
        "    optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate,\n",
        "                                 weight_decay=hparams.weight_decay)\n",
        "\n",
        "    if hparams.fp16_run:\n",
        "        from apex import amp\n",
        "        model, optimizer = amp.initialize(\n",
        "            model, optimizer, opt_level='O2')\n",
        "\n",
        "    if hparams.distributed_run:\n",
        "        model = apply_gradient_allreduce(model)\n",
        "\n",
        "    criterion = Tacotron2Loss()\n",
        "\n",
        "    logger = prepare_directories_and_logger(\n",
        "        output_directory, log_directory, rank)\n",
        "\n",
        "    train_loader, valset, collate_fn = prepare_dataloaders(hparams)\n",
        "\n",
        "    # Load checkpoint if one exists\n",
        "    iteration = 0\n",
        "    epoch_offset = 0\n",
        "    if checkpoint_path is not None or os.path.isfile(checkpoint_path):\n",
        "        if warm_start:\n",
        "            model = warm_start_model(\n",
        "                checkpoint_path, model, hparams.ignore_layers)\n",
        "        else:\n",
        "            model, optimizer, _learning_rate, iteration = load_checkpoint(\n",
        "                checkpoint_path, model, optimizer)\n",
        "            if hparams.use_saved_learning_rate:\n",
        "                learning_rate = _learning_rate\n",
        "            iteration += 1  # next iteration is iteration + 1\n",
        "            epoch_offset = max(0, int(iteration / len(train_loader)))\n",
        "    else:\n",
        "      os.path.isfile(\"pretrained_model\")\n",
        "      download_from_google_drive(\"1c5ZTuT7J08wLUoVZ2KkUs_VdZuJ86ZqA\",\"pretrained_model\")\n",
        "      model = warm_start_model(\"pretrained_model\", model, hparams.ignore_layers)\n",
        "      # download LJSpeech pretrained model if no checkpoint already exists\n",
        "    \n",
        "    start_eposh = time.perf_counter()\n",
        "    learning_rate = 0.0\n",
        "    model.train()\n",
        "    is_overflow = False\n",
        "    # ================ MAIN TRAINNIG LOOP! ===================\n",
        "    for epoch in tqdm(range(epoch_offset, hparams.epochs)):\n",
        "        print(\"\\nStarting Epoch: {} Iteration: {}\".format(epoch, iteration))\n",
        "        start_eposh = time.perf_counter() # eposh is russian, not a typo\n",
        "        for i, batch in tqdm(enumerate(train_loader), total=len(train_loader)):\n",
        "            start = time.perf_counter()\n",
        "            if iteration < hparams.decay_start: learning_rate = hparams.A_\n",
        "            else: iteration_adjusted = iteration - hparams.decay_start; learning_rate = (hparams.A_*(e**(-iteration_adjusted/hparams.B_))) + hparams.C_\n",
        "            learning_rate = max(hparams.min_learning_rate, learning_rate) # output the largest number\n",
        "            for param_group in optimizer.param_groups:\n",
        "                param_group['lr'] = learning_rate\n",
        "\n",
        "            model.zero_grad()\n",
        "            x, y = model.parse_batch(batch)\n",
        "            y_pred = model(x)\n",
        "\n",
        "            loss = criterion(y_pred, y)\n",
        "            if hparams.distributed_run:\n",
        "                reduced_loss = reduce_tensor(loss.data, n_gpus).item()\n",
        "            else:\n",
        "                reduced_loss = loss.item()\n",
        "            if hparams.fp16_run:\n",
        "                with amp.scale_loss(loss, optimizer) as scaled_loss:\n",
        "                    scaled_loss.backward()\n",
        "            else:\n",
        "                loss.backward()\n",
        "\n",
        "            if hparams.fp16_run:\n",
        "                grad_norm = torch.nn.utils.clip_grad_norm_(\n",
        "                    amp.master_params(optimizer), hparams.grad_clip_thresh)\n",
        "                is_overflow = math.isnan(grad_norm)\n",
        "            else:\n",
        "                grad_norm = torch.nn.utils.clip_grad_norm_(\n",
        "                    model.parameters(), hparams.grad_clip_thresh)\n",
        "\n",
        "            optimizer.step()\n",
        "\n",
        "            if not is_overflow and rank == 0:\n",
        "                duration = time.perf_counter() - start\n",
        "                logger.log_training(\n",
        "                    reduced_loss, grad_norm, learning_rate, duration, iteration)\n",
        "                #print(\"Batch {} loss {:.6f} Grad Norm {:.6f} Time {:.6f}\".format(iteration, reduced_loss, grad_norm, duration), end='\\r', flush=True)\n",
        "\n",
        "            iteration += 1\n",
        "        validate(model, criterion, valset, iteration,\n",
        "                 hparams.batch_size, n_gpus, collate_fn, logger,\n",
        "                 hparams.distributed_run, rank, epoch, start_eposh, learning_rate)\n",
        "        save_checkpoint(model, optimizer, learning_rate, iteration, checkpoint_path)\n",
        "        if log_directory2 != None:\n",
        "            copy_tree(log_directory, log_directory2)\n",
        "def check_dataset(hparams):\n",
        "    from utils import load_wav_to_torch, load_filepaths_and_text\n",
        "    import os\n",
        "    import numpy as np\n",
        "    def check_arr(filelist_arr):\n",
        "        for i, file in enumerate(filelist_arr):\n",
        "            if len(file) > 2:\n",
        "                print(\"|\".join(file), \"\\nhas multiple '|', this may not be an error.\")\n",
        "            if hparams.load_mel_from_disk and '.wav' in file[0]:\n",
        "                print(\"[WARNING]\", file[0], \" in filelist while expecting .npy .\")\n",
        "            else:\n",
        "                if not hparams.load_mel_from_disk and '.npy' in file[0]:\n",
        "                    print(\"[WARNING]\", file[0], \" in filelist while expecting .wav .\")\n",
        "            if (not os.path.exists(file[0])):\n",
        "                print(\"|\".join(file), \"\\n[WARNING] does not exist.\")\n",
        "            if len(file[1]) < 3:\n",
        "                print(\"|\".join(file), \"\\n[info] has no/very little text.\")\n",
        "            if not ((file[1].strip())[-1] in r\"!?,.;:\"):\n",
        "                print(\"|\".join(file), \"\\n[info] has no ending punctuation.\")\n",
        "            mel_length = 1\n",
        "            if hparams.load_mel_from_disk and '.npy' in file[0]:\n",
        "                melspec = torch.from_numpy(np.load(file[0], allow_pickle=True))\n",
        "                mel_length = melspec.shape[1]\n",
        "            if mel_length == 0:\n",
        "                print(\"|\".join(file), \"\\n[WARNING] has 0 duration.\")\n",
        "    print(\"Checking Training Files\")\n",
        "    audiopaths_and_text = load_filepaths_and_text(hparams.training_files) # get split lines from training_files text file.\n",
        "    check_arr(audiopaths_and_text)\n",
        "    print(\"Checking Validation Files\")\n",
        "    audiopaths_and_text = load_filepaths_and_text(hparams.validation_files) # get split lines from validation_files text file.\n",
        "    check_arr(audiopaths_and_text)\n",
        "    print(\"Finished Checking\")\n",
        "\n",
        "# ---- Replace .wav with .npy in filelists ----\n",
        "!sed -i -- 's,.wav|,.npy|,g' filelists/*.txt\n",
        "# ---- Replace .wav with .npy in filelists ----\n",
        "\n",
        "warm_start=False#sorry bout that\n",
        "n_gpus=1\n",
        "rank=0\n",
        "group_name=None\n",
        "\n",
        "# ---- DEFAULT PARAMETERS DEFINED HERE ----\n",
        "hparams = create_hparams()\n",
        "model_filename = 'current_model'\n",
        "hparams.training_files = \"filelists/list.txt\"\n",
        "hparams.validation_files = \"filelists/list.txt\"\n",
        "#hparams.use_mmi=True,          # not used in this notebook\n",
        "#hparams.use_gaf=True,          # not used in this notebook\n",
        "#hparams.max_gaf=0.5,           # not used in this notebook\n",
        "#hparams.drop_frame_rate = 0.2  # not used in this notebook\n",
        "hparams.p_attention_dropout=0.1\n",
        "hparams.p_decoder_dropout=0.1\n",
        "hparams.decay_start = 15000\n",
        "hparams.A_ = 5e-4\n",
        "hparams.B_ = 8000\n",
        "hparams.C_ = 0\n",
        "hparams.min_learning_rate = 1e-5\n",
        "generate_mels = True\n",
        "hparams.show_alignments = True\n",
        "alignment_graph_height = 600\n",
        "alignment_graph_width = 1000\n",
        "hparams.batch_size = 32\n",
        "hparams.load_mel_from_disk = True\n",
        "hparams.ignore_layers = []\n",
        "hparams.epochs = 10000\n",
        "torch.backends.cudnn.enabled = hparams.cudnn_enabled\n",
        "torch.backends.cudnn.benchmark = hparams.cudnn_benchmark\n",
        "output_directory = '/content/drive/My Drive/colab/outdir' # Location to save Checkpoints\n",
        "log_directory = '/content/tacotron2/logs' # Location to save Log files locally\n",
        "log_directory2 = '/content/drive/My Drive/colab/logs' # Location to copy log files (done at the end of each epoch to cut down on I/O)\n",
        "checkpoint_path = output_directory+(r'/')+model_filename"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wv62yzewMKui"
      },
      "source": [
        "This is where you put the name of the model (usually the name of the person's voice you're cloning)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 4,
      "metadata": {
        "cellView": "form",
        "id": "mWF5VuGELRey",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "#@title Name the model\n",
        "\n",
        "model_filename = \"LJ\" #@param {type:\"string\"}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-n3e8PDeMAWi"
      },
      "source": [
        "Replace \"PUTFILENAMEHERE\" with the name of your text file."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 5,
      "metadata": {
        "id": "zyP8e7TfLtmX",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "hparams.training_files = \"filelists/list.txt\"\n",
        "hparams.validation_files = \"filelists/list.txt\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ut6p3YmSNYY6"
      },
      "source": [
        "### Set the parameters\n",
        "\n",
        "You can play around with these if you want."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 6,
      "metadata": {
        "id": "dpm433-rMfKt",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": [
        "# hparams to Tune\n",
        "\n",
        "#These two are the most important\n",
        "hparams.batch_size = 32 # Controls how fast the model trains. Don't set this too high, or else it will GPU will OOM (out of memory). 30-ish is usually a good number if you have a bigger dataset. If the number of audio files is more than/about the same as this number, it won't train properly, and you won't be able to use it.\n",
        "hparams.epochs = 500 # Maxmimum epochs (number of times the AI looks through the dataset) to train\n",
        "\n",
        "#The rest aren't that important\n",
        "hparams.p_attention_dropout=0.1\n",
        "hparams.p_decoder_dropout=0.1\n",
        "hparams.decay_start = 15000         # wait till decay_start to start decaying learning rate\n",
        "hparams.A_ = 5e-4                   # Start/Max Learning Rate\n",
        "hparams.B_ = 8000                   # Decay Rate\n",
        "hparams.C_ = 0                      # Shift learning rate equation by this value\n",
        "hparams.min_learning_rate = 1e-5    # Min Learning Rate\n",
        "generate_mels = True # Don't change\n",
        "hparams.show_alignments = True\n",
        "alignment_graph_height = 600\n",
        "alignment_graph_width = 1000\n",
        "hparams.load_mel_from_disk = True\n",
        "hparams.ignore_layers = [] # Layers to reset (None by default, other than foreign languages this param can be ignored)\n",
        "\n",
        "torch.backends.cudnn.enabled = hparams.cudnn_enabled\n",
        "torch.backends.cudnn.benchmark = hparams.cudnn_benchmark\n",
        "output_directory = '/content/drive/My Drive/colab/outdir' # Location to save Checkpoints\n",
        "log_directory = '/content/tacotron2/logs' # Location to save Log files locally\n",
        "log_directory2 = '/content/drive/My Drive/colab/logs' # Location to copy log files (done at the end of each epoch to cut down on I/O)\n",
        "checkpoint_path = output_directory+(r'/')+model_filename"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "XbcqKVlNNxxV"
      },
      "source": [
        "### Create MEL spectrograms\n",
        "\n",
        "The MEL spectrograms are the input for the AI."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 8,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 66,
          "referenced_widgets": [
            "9a906d996ca647e79321956289fad9ff",
            "335535bd785b4e96b4dfab8bd6a95436",
            "85d2ef70d28b412e9efa1302ed9c8c0c",
            "6154427d5c1a4afeab65ba1f18fcd29b",
            "81569de6fdb844aea41e19381181c288",
            "e9396ddc55724528b6ebf53b6ab6bb22",
            "d6612ee4facd48cbaa6e0ff6cfcc6753",
            "e27d05bc589844e7a5753c3d751362c6",
            "0ad58687958a49e181959dce9dc6c625",
            "b31dc29e0efc46a085e2db959b448e14",
            "ae7616d642184d04870a4743bb94fb25"
          ]
        },
        "id": "JiuUJH_ANSmv",
        "outputId": "58033662-9c7b-4d01-b717-b408ee0ee35f",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Generating Mels\n"
          ]
        },
        {
          "data": {
            "application/vnd.jupyter.widget-view+json": {
              "model_id": "9a906d996ca647e79321956289fad9ff",
              "version_major": 2,
              "version_minor": 0
            },
            "text/plain": [
              "  0%|          | 0/13100 [00:00<?, ?it/s]"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "if generate_mels:\n",
        "    create_mels()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "OMKk2asHOAKh"
      },
      "source": [
        "### Check data\n",
        "\n",
        "This cell will go through the dataset one more time to see if anything's wrong before training starts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 9,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "rJ08eb1eOFau",
        "outputId": "ec51579f-df21-4ede-8e01-f0d6d4ac051a",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "Checking Training Files\n",
            "Checking Validation Files\n",
            "Finished Checking\n"
          ]
        }
      ],
      "source": [
        "check_dataset(hparams)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0LDuVYavOPij"
      },
      "source": [
        "# Start training!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wpe88HwnOT2p"
      },
      "source": [
        "This next cell will begin training the model!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": 10,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "id": "oq0cH7AtOEAx",
        "outputId": "cfe33d6e-5b3b-4536-9962-38f41e278992",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "FP16 Run: False\n",
            "Dynamic Loss Scaling: True\n",
            "Distributed Run: False\n",
            "cuDNN Enabled: True\n",
            "cuDNN Benchmark: False\n",
            "Loading checkpoint '/content/drive/My Drive/colab/outdir/LJ'\n"
          ]
        },
        {
          "ename": "FileNotFoundError",
          "evalue": "ignored",
          "output_type": "error",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-10-8836047e5d65>\u001b[0m in \u001b[0;36m<cell line: 7>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      5\u001b[0m \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'cuDNN Benchmark:'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhparams\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcudnn_benchmark\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 7\u001b[0;31m train(output_directory, log_directory, checkpoint_path,\n\u001b[0m\u001b[1;32m      8\u001b[0m       warm_start, n_gpus, rank, group_name, hparams, log_directory2)\n",
            "\u001b[0;32m<ipython-input-3-1a8318b5beab>\u001b[0m in \u001b[0;36mtrain\u001b[0;34m(output_directory, log_directory, checkpoint_path, warm_start, n_gpus, rank, group_name, hparams, log_directory2)\u001b[0m\n\u001b[1;32m    268\u001b[0m                 checkpoint_path, model, hparams.ignore_layers)\n\u001b[1;32m    269\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 270\u001b[0;31m             model, optimizer, _learning_rate, iteration = load_checkpoint(\n\u001b[0m\u001b[1;32m    271\u001b[0m                 checkpoint_path, model, optimizer)\n\u001b[1;32m    272\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhparams\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0muse_saved_learning_rate\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-3-1a8318b5beab>\u001b[0m in \u001b[0;36mload_checkpoint\u001b[0;34m(checkpoint_path, model, optimizer)\u001b[0m\n\u001b[1;32m    150\u001b[0m     \u001b[0;31m# assert os.path.isfile(checkpoint_path)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    151\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Loading checkpoint '{}'\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_path\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 152\u001b[0;31m     \u001b[0mcheckpoint_dict\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_path\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmap_location\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'cpu'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    153\u001b[0m     \u001b[0mmodel\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'state_dict'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    154\u001b[0m     \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mload_state_dict\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mcheckpoint_dict\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'optimizer'\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/serialization.py\u001b[0m in \u001b[0;36mload\u001b[0;34m(f, map_location, pickle_module, weights_only, **pickle_load_args)\u001b[0m\n\u001b[1;32m    789\u001b[0m         \u001b[0mpickle_load_args\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m'encoding'\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'utf-8'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    790\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 791\u001b[0;31m     \u001b[0;32mwith\u001b[0m \u001b[0m_open_file_like\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'rb'\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0mopened_file\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    792\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0m_is_zipfile\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopened_file\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    793\u001b[0m             \u001b[0;31m# The zipfile reader is going to advance the current file position.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/serialization.py\u001b[0m in \u001b[0;36m_open_file_like\u001b[0;34m(name_or_buffer, mode)\u001b[0m\n\u001b[1;32m    269\u001b[0m \u001b[0;32mdef\u001b[0m \u001b[0m_open_file_like\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    270\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0m_is_path\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_buffer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 271\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0m_open_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname_or_buffer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    272\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    273\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;34m'w'\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.9/dist-packages/torch/serialization.py\u001b[0m in \u001b[0;36m__init__\u001b[0;34m(self, name, mode)\u001b[0m\n\u001b[1;32m    250\u001b[0m \u001b[0;32mclass\u001b[0m \u001b[0m_open_file\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0m_opener\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    251\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 252\u001b[0;31m         \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__init__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mopen\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mname\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mmode\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    253\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    254\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__exit__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: '/content/drive/My Drive/colab/outdir/LJ'"
          ]
        }
      ],
      "source": [
        "print('FP16 Run:', hparams.fp16_run)\n",
        "print('Dynamic Loss Scaling:', hparams.dynamic_loss_scaling)\n",
        "print('Distributed Run:', hparams.distributed_run)\n",
        "print('cuDNN Enabled:', hparams.cudnn_enabled)\n",
        "print('cuDNN Benchmark:', hparams.cudnn_benchmark)\n",
        "\n",
        "train(output_directory, log_directory, checkpoint_path,\n",
        "      warm_start, n_gpus, rank, group_name, hparams, log_directory2)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZycbYtm54k1M",
        "vscode": {
          "languageId": "python"
        }
      },
      "outputs": [],
      "source": []
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "widgets": {
      "application/vnd.jupyter.widget-state+json": {
        "0ad58687958a49e181959dce9dc6c625": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "ProgressStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "ProgressStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "bar_color": null,
            "description_width": ""
          }
        },
        "335535bd785b4e96b4dfab8bd6a95436": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e9396ddc55724528b6ebf53b6ab6bb22",
            "placeholder": "​",
            "style": "IPY_MODEL_d6612ee4facd48cbaa6e0ff6cfcc6753",
            "value": "100%"
          }
        },
        "6154427d5c1a4afeab65ba1f18fcd29b": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HTMLModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HTMLModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HTMLView",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_b31dc29e0efc46a085e2db959b448e14",
            "placeholder": "​",
            "style": "IPY_MODEL_ae7616d642184d04870a4743bb94fb25",
            "value": " 13100/13100 [06:10&lt;00:00, 44.14it/s]"
          }
        },
        "81569de6fdb844aea41e19381181c288": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "85d2ef70d28b412e9efa1302ed9c8c0c": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "FloatProgressModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "FloatProgressModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "ProgressView",
            "bar_style": "success",
            "description": "",
            "description_tooltip": null,
            "layout": "IPY_MODEL_e27d05bc589844e7a5753c3d751362c6",
            "max": 13100,
            "min": 0,
            "orientation": "horizontal",
            "style": "IPY_MODEL_0ad58687958a49e181959dce9dc6c625",
            "value": 13100
          }
        },
        "9a906d996ca647e79321956289fad9ff": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "HBoxModel",
          "state": {
            "_dom_classes": [],
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "HBoxModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/controls",
            "_view_module_version": "1.5.0",
            "_view_name": "HBoxView",
            "box_style": "",
            "children": [
              "IPY_MODEL_335535bd785b4e96b4dfab8bd6a95436",
              "IPY_MODEL_85d2ef70d28b412e9efa1302ed9c8c0c",
              "IPY_MODEL_6154427d5c1a4afeab65ba1f18fcd29b"
            ],
            "layout": "IPY_MODEL_81569de6fdb844aea41e19381181c288"
          }
        },
        "ae7616d642184d04870a4743bb94fb25": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "b31dc29e0efc46a085e2db959b448e14": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "d6612ee4facd48cbaa6e0ff6cfcc6753": {
          "model_module": "@jupyter-widgets/controls",
          "model_module_version": "1.5.0",
          "model_name": "DescriptionStyleModel",
          "state": {
            "_model_module": "@jupyter-widgets/controls",
            "_model_module_version": "1.5.0",
            "_model_name": "DescriptionStyleModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "StyleView",
            "description_width": ""
          }
        },
        "e27d05bc589844e7a5753c3d751362c6": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        },
        "e9396ddc55724528b6ebf53b6ab6bb22": {
          "model_module": "@jupyter-widgets/base",
          "model_module_version": "1.2.0",
          "model_name": "LayoutModel",
          "state": {
            "_model_module": "@jupyter-widgets/base",
            "_model_module_version": "1.2.0",
            "_model_name": "LayoutModel",
            "_view_count": null,
            "_view_module": "@jupyter-widgets/base",
            "_view_module_version": "1.2.0",
            "_view_name": "LayoutView",
            "align_content": null,
            "align_items": null,
            "align_self": null,
            "border": null,
            "bottom": null,
            "display": null,
            "flex": null,
            "flex_flow": null,
            "grid_area": null,
            "grid_auto_columns": null,
            "grid_auto_flow": null,
            "grid_auto_rows": null,
            "grid_column": null,
            "grid_gap": null,
            "grid_row": null,
            "grid_template_areas": null,
            "grid_template_columns": null,
            "grid_template_rows": null,
            "height": null,
            "justify_content": null,
            "justify_items": null,
            "left": null,
            "margin": null,
            "max_height": null,
            "max_width": null,
            "min_height": null,
            "min_width": null,
            "object_fit": null,
            "object_position": null,
            "order": null,
            "overflow": null,
            "overflow_x": null,
            "overflow_y": null,
            "padding": null,
            "right": null,
            "top": null,
            "visibility": null,
            "width": null
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}