20200902_Hackathon_Notes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# iR+ Hackathon notes\n",
    "\n",
    "**Dates**: 2020/09/02 - 2020/09/04\n",
    "\n",
    "[**Hackathon Google Drive Folder**](https://drive.google.com/drive/folders/1C0ExnBbHFQH9SQrFIZ6J1Bn5Ify5mx0P)\n",
    "\n",
    "**Tool list**: [Hackathon_tools (Google Drive)](https://docs.google.com/spreadsheets/d/1gX--D6XY4JwbkoWo1hoFmZ7-IdGAv3d95aZRHcDjdGs/edit#gid=0)  \n",
    "**Presentation**: [ir+ Hackathon.pptx](./iR+_Hackathon.pptx) ([Google drive](https://drive.google.com/file/d/1pZOegzTnZMBP3LaFJ5HvEUmlCwzaAD7t/view?usp=sharing))\n",
    "\n",
    "**Videos**:\n",
    "- [Day 1](./iReceptorPlusHackathon20200902.mp4) ([Google drive](https://drive.google.com/file/d/1pM0zgvgKWtcwBlgGrBMDWQBivxTk22ju/view))\n",
    "- [Day 2](./iReceptorPlusHackathon20200902.mp4) ([No direct link to Google drive, see Hackathon folder.](https://drive.google.com/drive/folders/1C0ExnBbHFQH9SQrFIZ6J1Bn5Ify5mx0P))\n",
    "\n",
    "**About Singularity**:\n",
    "- [User guide](https://sylabs.io/guides/3.6/user-guide/)\n",
    "- [Examples](https://github.com/sylabs/examples)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Environment Setup\n",
    "\n",
    "Setup your accounts and machine to use during the hackathon\n",
    "\n",
    "### Accounts\n",
    "- GitHub account\n",
    "        We will need to have access to the ireceptor-plus/tapis repository (this is where the first example is).\n",
    "- VDJ account\n",
    "        This will be required to access the singularity and data files, and to authenticate and authorize against the HPC (lonestar 5 at U.Texas: irplus-ls5.tacc.utexas.edu).\n",
    "        \n",
    "### Software\n",
    "- Download and install Git from the [Git official site](https://git-scm.com/). Check one pf the [GUI for windows](https://git-scm.com/download/gui/windows).\n",
    "- Download and install Docker Desktop from the [Docker official site](https://www.docker.com/products/docker-desktop)\n",
    "- Some editor to edit the files ([notepad++](https://notepad-plus-plus.org/), [Brackets](http://brackets.io/), [VSCode](https://code.visualstudio.com/), ...)\n",
    "\n",
    "### Filesystem\n",
    "We will need a filesystem folder for the Hackathon, let call it _20200902_hackathon_.  \n",
    "Inside the hackathon folder we will need two folders, both folders (and their names) will be required for running the docker image:\n",
    "- **irplus**\n",
    "        holds the clone of the github repository.\n",
    "- **tapis-token**\n",
    "        Used to give persistence to the auth token that will be used by tapis.\n",
    "\n",
    "Open a __command line__ or a __Windows powershell__ in a folder where you want to add your hackathon area and execute the following instruction to create the filesystem structure:\n",
    "> mkdir 20200902_hackathon; cd 20200902_hackathon\\; mkdir irplus; mkdir tapis-token"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Docker and GIT Setup\n",
    "\n",
    "We need to get the docker image and clone the git repository.\n",
    "\n",
    "### Docker image\n",
    "\n",
    "The Docker image can be pulled from the docker registry. Using the opened __command line__ or __Windows powershell__ execute:\n",
    "\n",
    "> docker pull ireceptorplus/tapis\n",
    "\n",
    "The output should be:\n",
    "\n",
    "        Using default tag: latest  \n",
    "        latest: Pulling from ireceptorplus/tapis  \n",
    "        Digest: sha256:6e66e16c9ddae44d37bc20d289eecbcc6bbd8b7f57ef97ad765c7e341f5d4428  \n",
    "        Status: Image is up to date for ireceptorplus/tapis:latest  \n",
    "        docker.io/ireceptorplus/tapis:latest  \n",
    "\n",
    "### GIT clone\n",
    "\n",
    "**Previous note**: We found that, on windows machines, the following error was thrown during the execution of the docker image. This is an error with the line endings and can be easily solved with the _git clone_  instruction.\n",
    "        \n",
    "        upload-bundle.sh: line 17: $'\\r': command not found\n",
    "\n",
    "To clone the ireceptor-plus/tapis repo, execute on the __command line__ or __Windows powershell__:  \n",
    "> git clone https://github.com/ireceptor-plus/tapis.git irplus --config core.autocrlf=input\n",
    "\n",
    "The output should be similar to:  \n",
    "\n",
    "        Cloning into 'irplus'...\n",
    "        remote: Enumerating objects: 128, done.\n",
    "        remote: Counting objects: 100% (128/128), done.\n",
    "        remote: Compressing objects: 100% (96/96), done.\n",
    "        Receiving objects8 (delta 43), reused 102 (delta 21), pack-reused 0eceiving objects:  42% (54/128), 3.04 MiB | 6.06\n",
    "        Receiving objects: 100% (128/128), 7.52 MiB | 6.06 MiB/s, done.\n",
    "        Resolving deltas: 100% (43/43), done.\n",
    "        \n",
    "The parameter '_--config core.autocrlf=input_' on the _clone_ instruction tells git to automatically change all **LF** (linux/macos) line endings to **CRLF** (windows) line endings. As it appears the same parameter will ensure that all **CRLF** are reverted to **LF** upon committing to GIT.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running Docker\n",
    "\n",
    "To start the docker image, use the opened __command line__ or __Windows powershell__ and execute: \n",
    "> docker run -v \\\\$PWD/tapis-token/mao:/root/.agave -v \\\\$PWD/irplus:/work -it ireceptorplus/tapis bash\n",
    "\n",
    "You will get a bash within the docker environment.\n",
    "\n",
    "Then to check if the filesystem mapping is correct you can list the contents of the _/work_ and _/root/.agave_ and check that it maps with the contents of the folders **irplus** and **tapis-token** previously created.\n",
    "\n",
    "**NOTE**: During docker execution changes in the contents of **irplus** and **tapis-token** folders reflect both ways (inside and outside docker). Thus we can edit the files in **irplus** outside the docker and use them inside the docker environment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tutorial - Toy application\n",
    "\n",
    "### Step 1 - Edit files\n",
    "\n",
    "In your phisical machine (__command line__, __Windows powershell__ or __windows explorer__). Change to folder _20200902_Hackathon\\irplus\\apps\\toy\\0.1\\ls5\\_\n",
    "\n",
    "> cd irplus\\apps\\toy\\0.1\\ls5\\\n",
    "\n",
    "Edit the files _toy.json_, _upload-bundle.sh_ and _test\\test-app.json_.  \n",
    "\n",
    "        Replace  all values **schristley** to your **username** (example **mao**)\n",
    "\n",
    "This will ensure that we use a unique directory to upload the files, and that will not override other user files.\n",
    "\n",
    "        \n",
    "### Step 2 - Authetication\n",
    "\n",
    "Within **docker bash**, initialize tapis authentication:\n",
    "\n",
    "> tapis auth init\n",
    "\n",
    "It will start an interactive process\n",
    "\n",
    "        +---------------+--------------------------------------+----------------------------------------+\n",
    "        |      Name     |             Description              |                  URL                   |\n",
    "        +---------------+--------------------------------------+----------------------------------------+\n",
    "        |      3dem     |             3dem Tenant              |         https://api.3dem.org/          |\n",
    "        |   agave.prod  |         Agave Public Tenant          |      https://public.agaveapi.co/       |\n",
    "        |  araport.org  |               Araport                |        https://api.araport.org/        |\n",
    "        |     bridge    |                Bridge                |     https://api.bridge.tacc.cloud/     |\n",
    "        |   designsafe  |              DesignSafe              |    https://agave.designsafe-ci.org/    |\n",
    "        |  iplantc.org  |         CyVerse Science APIs         |       https://agave.iplantc.org/       |\n",
    "        |      irec     |              iReceptor               | https://irec.tenants.prod.tacc.cloud/  |\n",
    "        |    portals    |            Portals Tenant            |  https://portals-api.tacc.utexas.edu/  |\n",
    "        |      sd2e     |             SD2E Tenant              |         https://api.sd2e.org/          |\n",
    "        |      sgci     | Science Gateways Community Institute |        https://sgci.tacc.cloud/        |\n",
    "        |   tacc.prod   |                 TACC                 |      https://api.tacc.utexas.edu/      |\n",
    "        | vdjserver.org |              VDJ Server              | https://vdj-agave-api.tacc.utexas.edu/ |\n",
    "        +---------------+--------------------------------------+----------------------------------------+\n",
    "        Enter a tenant name [tacc.prod]:\n",
    "\n",
    "> vdjserver.org\n",
    "\n",
    "        vdjserver.org username: \n",
    "\n",
    "> <username>\n",
    "    \n",
    "        vdjserver.org password for mao:\n",
    "\n",
    "> <password>\n",
    "\n",
    "For the next questions just hit enter.\n",
    "\n",
    "        Container registry access:\n",
    "        --------------------------\n",
    "        Registry Url [https://index.docker.io]:\n",
    "        Registry Username:\n",
    "        Registry Password:\n",
    "        Registry Namespace:\n",
    "\n",
    "        Git server access:\n",
    "        ------------------\n",
    "        Git Username:\n",
    "        Learn about github.com personal access tokens:\n",
    "        https://help.github.com/en/github/authenticating-to-github/creating-a-personal-access-token-for-the-command-line\n",
    "        Git Token:\n",
    "        Git Namespace:\n",
    "        +--------------------+-------------------------------------+\n",
    "        | Field              | Value                               |\n",
    "        +--------------------+-------------------------------------+\n",
    "        | tenant_id          | vdjserver.org                       |\n",
    "        | username           | <username>                          |\n",
    "        | client_name        | _cli-vdjserver.org-mao-8f4a1317b803 |\n",
    "        | api_key            | UdMXETMq6KXutzHlwOIsEMISU4ca        |\n",
    "        | access_token       | 949f61032efea2864c476f6a7bd1d5      |\n",
    "        | expires_at         | Thu Sep  3 15:06:42 2020            |\n",
    "        | verify             | True                                |\n",
    "        | registry_url       | https://index.docker.io             |\n",
    "        | registry_username  |                                     |\n",
    "        | registry_password  |                                     |\n",
    "        | registry_namespace |                                     |\n",
    "        | git_username       |                                     |\n",
    "        | git_token          |                                     |\n",
    "        | git_namespace      |                                     |\n",
    "        +--------------------+-------------------------------------+\n",
    "\n",
    "\n",
    "**NOTE**: To renew the tokens use (a token has a life of ~ one hour):\n",
    "\n",
    "> tapis auth tokens refresh\n",
    "\n",
    "To create a new token:  \n",
    "\n",
    "> tapis auth token create\n",
    "\n",
    "\n",
    "To show the token:\n",
    "    \n",
    "> tapis auth show\n",
    "    \n",
    "        +---------------+----------------------------------+\n",
    "        | Field         | Value                            |\n",
    "        +---------------+----------------------------------+\n",
    "        | tenant_id     | vdjserver.org                    |\n",
    "        | username      | mao                              |\n",
    "        | api_key       | UdMXETMq6KXutzHlwOIsEMISU4ca     |\n",
    "        | access_token  | 949f61032efea2864c476f6a7bd1d5   |\n",
    "        | expires_at    | Thu Sep  3 15:06:42 2020         |\n",
    "        | refresh_token | a365aac7605df77b9f6886b584eb7525 |\n",
    "        +---------------+----------------------------------+\n",
    "    \n",
    "### Step 3 - (if required) prior setup \n",
    " \n",
    "List all available systems  \n",
    "    \n",
    "> tapis systems list\n",
    "    \n",
    "        +-----------------------------+--------------------------+-----------+---------+\n",
    "        | id                          | name                     | type      | default |\n",
    "        +-----------------------------+--------------------------+-----------+---------+\n",
    "        | ls5.tacc.utexas.edu         | Lonestar 5 Supercomputer | EXECUTION | False   |\n",
    "        | data.vdjserver.org          | VDJData                  | STORAGE   | True    |\n",
    "        | irplus-ls5.tacc.utexas.edu  | Lonestar 5 Supercomputer | EXECUTION | False   |\n",
    "        | stampede2.tacc.utexas.edu   | Stampede2 Supercomputer  | EXECUTION | False   |\n",
    "        | stampede.tacc.utexas.edu    | Stampede Supercomputer   | EXECUTION | False   |\n",
    "        | vdj-exec-02.tacc.utexas.edu | VDJ Exec 02              | EXECUTION | False   |\n",
    "        | lonestar.tacc.utexas.edu    | Lonestar 4 Supercomputer | EXECUTION | False   |\n",
    "        | vdj-exec-01.tacc.utexas.edu | VDJ Exec 01              | EXECUTION | False   |\n",
    "        | sftp.vdjserver.org          | SFTP Dev System          | STORAGE   | False   |\n",
    "        +-----------------------------+--------------------------+-----------+---------+\n",
    "    \n",
    "List files and folders  \n",
    "\n",
    "> tapis files list agave:///irplus\n",
    "    \n",
    "        +--------+----------------+--------+\n",
    "        | name   | lastModified   | length |\n",
    "        +--------+----------------+--------+\n",
    "        | apps   | 16 hours ago   |   4096 |\n",
    "        | data   | 18 hours ago   |   4096 |\n",
    "        | images | 22 minutes ago |   4096 |\n",
    "        +--------+----------------+--------+\n",
    "\n",
    "Folders meaning:\n",
    "- apps: Application assets (tapis concept=definition of inputs and parameters, script and executable files that can be ran as a tapis execution.\n",
    "- data: Available data\n",
    "- images: Put code on the super computer (singularity images). A image is one of the files that compose an APP.\n",
    "\n",
    "Upload files:  \n",
    "\n",
    "> tapis files upload ...\n",
    "    \n",
    "List all apps\n",
    "    \n",
    "> tapis apps list\n",
    "    \n",
    "\n",
    "\n",
    "### step 4 - Upload files\n",
    "\n",
    "Within docker, change directory to _/work/apps/toy/0.1/ls5_:\n",
    "\n",
    "> cd /work/apps/toy/0.1/ls5\n",
    "\n",
    "Run _upload-bundle.sh_\n",
    "\n",
    "> bash upload-bundle.sh\n",
    "    \n",
    "Output:\n",
    "\n",
    "        +--------------+-------+\n",
    "        | Field        | Value |\n",
    "        +--------------+-------+\n",
    "        | deleted      | 0     |\n",
    "        | skipped      | 1     |\n",
    "        | warnings     | 1     |\n",
    "        | elapsed_msec | 0     |\n",
    "        +--------------+-------+\n",
    "        +--------------+--------------------------------------+\n",
    "        | Field        | Value                                |\n",
    "        +--------------+--------------------------------------+\n",
    "        | name         | toy                                  |\n",
    "        | uuid         | 713570405814440426-242ac112-0001-002 |\n",
    "        | owner        | vdj                                  |\n",
    "        | path         | /irplus/apps/toy                     |\n",
    "        | lastModified | 2020-09-03T09:41:42.222-05:00        |\n",
    "        | source       | None                                 |\n",
    "        | status       | STAGING_COMPLETED                    |\n",
    "        | nativeFormat | dir                                  |\n",
    "        | systemId     | data.vdjserver.org                   |\n",
    "        +--------------+--------------------------------------+\n",
    "        +--------------+---------------------------------------+\n",
    "        | Field        | Value                                 |\n",
    "        +--------------+---------------------------------------+\n",
    "        | name         | mao                                   |\n",
    "        | uuid         | 8020383441663815190-242ac112-0001-002 |\n",
    "        | owner        | mao                                   |\n",
    "        | path         | /irplus/apps/toy/mao                  |\n",
    "        | lastModified | 2020-09-03T09:41:45.735-05:00         |\n",
    "        | source       | None                                  |\n",
    "        | status       | STAGING_COMPLETED                     |\n",
    "        | nativeFormat | dir                                   |\n",
    "        | systemId     | data.vdjserver.org                    |\n",
    "        +--------------+---------------------------------------+\n",
    "        +--------------+---------------------------------------+\n",
    "        | Field        | Value                                 |\n",
    "        +--------------+---------------------------------------+\n",
    "        | name         | ls5                                   |\n",
    "        | uuid         | 7944596991379107350-242ac112-0001-002 |\n",
    "        | owner        | mao                                   |\n",
    "        | path         | /irplus/apps/toy/mao/ls5              |\n",
    "        | lastModified | 2020-09-03T09:41:54.278-05:00         |\n",
    "        | source       | None                                  |\n",
    "        | status       | STAGING_COMPLETED                     |\n",
    "        | nativeFormat | dir                                   |\n",
    "        | systemId     | data.vdjserver.org                    |\n",
    "        +--------------+---------------------------------------+\n",
    "        +--------------+---------------------------------------+\n",
    "        | Field        | Value                                 |\n",
    "        +--------------+---------------------------------------+\n",
    "        | name         | test                                  |\n",
    "        | uuid         | 7782633774646947350-242ac112-0001-002 |\n",
    "        | owner        | mao                                   |\n",
    "        | path         | /irplus/apps/toy/mao/ls5/test         |\n",
    "        | lastModified | 2020-09-03T09:41:57.699-05:00         |\n",
    "        | source       | None                                  |\n",
    "        | status       | STAGING_COMPLETED                     |\n",
    "        | nativeFormat | dir                                   |\n",
    "        | systemId     | data.vdjserver.org                    |\n",
    "        +--------------+---------------------------------------+\n",
    "        +-------------------+---------+\n",
    "        | Field             | Value   |\n",
    "        +-------------------+---------+\n",
    "        | uploaded          | 1       |\n",
    "        | skipped           | 0       |\n",
    "        | messages          | 0       |\n",
    "        | bytes_transferred | 1.41 kB |\n",
    "        | elapsed_sec       | 1       |\n",
    "        +-------------------+---------+\n",
    "        +-------------------+---------+\n",
    "        | Field             | Value   |\n",
    "        +-------------------+---------+\n",
    "        | uploaded          | 1       |\n",
    "        | skipped           | 0       |\n",
    "        | messages          | 0       |\n",
    "        | bytes_transferred | 3.48 kB |\n",
    "        | elapsed_sec       | 1       |\n",
    "        +-------------------+---------+\n",
    "        +-------------------+---------+\n",
    "        | Field             | Value   |\n",
    "        +-------------------+---------+\n",
    "        | uploaded          | 1       |\n",
    "        | skipped           | 0       |\n",
    "        | messages          | 0       |\n",
    "        | bytes_transferred | 3.25 kB |\n",
    "        | elapsed_sec       | 1       |\n",
    "        +-------------------+---------+\n",
    "        +---------------+----------------+--------+\n",
    "        | name          | lastModified   | length |\n",
    "        +---------------+----------------+--------+\n",
    "        | test          | 13 seconds ago |   4096 |\n",
    "        | toy.json      | just now       |   3568 |\n",
    "        | toy.sh        | just now       |   1442 |\n",
    "        | toy_common.sh | just now       |   3331 |\n",
    "        +---------------+----------------+--------+\n",
    "        +-------------------+------------+\n",
    "        | Field             | Value      |\n",
    "        +-------------------+------------+\n",
    "        | uploaded          | 1          |\n",
    "        | skipped           | 0          |\n",
    "        | messages          | 0          |\n",
    "        | bytes_transferred | 0.00 bytes |\n",
    "        | elapsed_sec       | 1          |\n",
    "        +-------------------+------------+\n",
    "        +-------------------+---------+\n",
    "        | Field             | Value   |\n",
    "        +-------------------+---------+\n",
    "        | uploaded          | 1       |\n",
    "        | skipped           | 0       |\n",
    "        | messages          | 0       |\n",
    "        | bytes_transferred | 3.75 MB |\n",
    "        | elapsed_sec       | 3       |\n",
    "        +-------------------+---------+\n",
    "        +---------+--------------+--------+\n",
    "        | name    | lastModified | length |\n",
    "        +---------+--------------+--------+\n",
    "        | test.sh | just now     |      0 |\n",
    "        +---------+--------------+--------+\n",
    "\n",
    "\n",
    "After running the upload script we should see a new folder <username> under agave:///irplus/apps/toy \n",
    "\n",
    "> tapis files list agave:///irplus/apps/toy\n",
    "    \n",
    "        +---------------+--------------+--------+\n",
    "        | name          | lastModified | length |\n",
    "        +---------------+--------------+--------+\n",
    "        | aurelienbzh   | a day ago    |   4096 |\n",
    "        | bcorrie       | a day ago    |   4096 |\n",
    "        | caschramm     | a day ago    |   4096 |\n",
    "        | edgar.s.silva | a day ago    |   4096 |\n",
    "        | franasa       | a day ago    |   4096 |\n",
    "        | grojas        | a day ago    |   4096 |\n",
    "        | mao           | an hour ago  |   4096 |\n",
    "        | schristley    | a day ago    |   4096 |\n",
    "        | sharikrish    | 7 hours ago  |   4096 |\n",
    "        +---------------+--------------+--------+\n",
    "\n",
    "> tapis files list agave:///irplus/apps/toy/mao/ls5\n",
    "    \n",
    "        +---------------+--------------+--------+\n",
    "        | name          | lastModified | length |\n",
    "        +---------------+--------------+--------+\n",
    "        | test          | an hour ago  |   4096 |\n",
    "        | toy.json      | an hour ago  |   3568 |\n",
    "        | toy.sh        | an hour ago  |   1442 |\n",
    "        | toy_common.sh | an hour ago  |   3331 |\n",
    "        +---------------+--------------+--------+\n",
    "    \n",
    "### Step 5 - Create the application\n",
    "    \n",
    "To create an application we will use the _tapis app_ command. To see the help type:  \n",
    "\n",
    ">  tapis apps create --help\n",
    "    \n",
    "It is using this command that we set the wrapper json definition file (in our case the _toy.json_ file).\n",
    "\n",
    "> less toy.json\n",
    "    \n",
    "\n",
    "Then we can create the application. The toy.json is a reference to the local (in docker) file, it is not necessary to be uploaded.\n",
    "    \n",
    "> tapis apps create -F toy.json\n",
    "\n",
    "        +--------------------------+---------------------------------------+\n",
    "        | Field                    | Value                                 |\n",
    "        +--------------------------+---------------------------------------+\n",
    "        | id                       | irplus-toy-ls5-mao-0.1                |\n",
    "        | name                     | irplus-toy-ls5-mao                    |\n",
    "        | version                  | 0.1                                   |\n",
    "        | revision                 | 2                                     |\n",
    "        | label                    | toy                                   |\n",
    "        | lastModified             | just now                              |\n",
    "        | shortDescription         | Toy app on Lonestar5                  |\n",
    "        | longDescription          | iReceptor+ toy app                    |\n",
    "        | owner                    | mao                                   |\n",
    "        | isPublic                 | False                                 |\n",
    "        | executionType            | HPC                                   |\n",
    "        | executionSystem          | irplus-ls5.tacc.utexas.edu            |\n",
    "        | deploymentSystem         | data.vdjserver.org                    |\n",
    "        | available                | True                                  |\n",
    "        | parallelism              | PARALLEL                              |\n",
    "        | defaultProcessorsPerNode | 40                                    |\n",
    "        | defaultMemoryPerNode     | 1                                     |\n",
    "        | defaultNodeCount         | 1                                     |\n",
    "        | defaultMaxRunTime        | None                                  |\n",
    "        | defaultQueue             | normal                                |\n",
    "        | helpURI                  | None                                  |\n",
    "        | deploymentPath           | /irplus/apps/toy/mao/ls5              |\n",
    "        | templatePath             | toy.sh                                |\n",
    "        | testPath                 | test/test.sh                          |\n",
    "        | checkpointable           | False                                 |\n",
    "        | uuid                     | 8883588656832844266-242ac117-0001-005 |\n",
    "        | icon                     | None                                  |\n",
    "        +--------------------------+---------------------------------------+\n",
    "\n",
    "If the create command runs ok, then the application will have an identifier, which is the name and the version attached to it. I this case the identifier is the value on the field id (above) _irplus-toy-ls5-mao-0.1_.\n",
    "With this value we can use the show command\n",
    "    \n",
    "> tapis appls show <appId>\n",
    "\n",
    "Results in the same table above.\n",
    "    \n",
    "We can list all applications with:\n",
    "    \n",
    "> tapis apps list\n",
    "    \n",
    "    \n",
    "        +------------------------------+-------------+\n",
    "        | id                           | label       |\n",
    "        +------------------------------+-------------+\n",
    "        | irplus-toy-ls5-mao-0.1       | toy         |\n",
    "        | irplus-singularity-ls5-0.1u1 | singularity |\n",
    "        | igblast-stampede2-1.14u2     | igblast     |\n",
    "        | repcalc-stampede2-1.0u5      | repcalc     |\n",
    "        | repcalc-stampede2-1.0u4      | repcalc     |\n",
    "        | vdj_pipe-stampede2-0.1.7u3   | vdj_pipe    |\n",
    "        | presto-stampede2-0.5u2       | None        |\n",
    "        | igblast-stampede2-1.14u1     | igblast     |\n",
    "        | presto-ls5-0.5.2u6           | None        |\n",
    "        +------------------------------+-------------+\n",
    " \n",
    "    \n",
    "### Step 6 - Submit a Job\n",
    "\n",
    "We have a test job (test-app.json) inside the _test_folder.\n",
    "\n",
    "\n",
    "Change to test folder:\n",
    "    \n",
    "> cd test\n",
    "\n",
    "See the test job app:\n",
    "    \n",
    "> less test-app.json\n",
    "    \n",
    "        {\n",
    "          \"name\":\"test_toy\",\n",
    "          \"appId\": \"irplus-toy-ls5-mao-0.1\",\n",
    "          \"batchQueue\": \"normal\",\n",
    "          \"maxRunTime\": \"01:00:00\",\n",
    "          \"nodeCount\": 1,\n",
    "          \"archive\": false,\n",
    "          \"archiveSystem\": \"data.vdjserver.org\",\n",
    "          \"inputs\": {\n",
    "            \"singularity_image\": \"agave://data.vdjserver.org//irplus/images/immcantation_suite-4.1.0.sif\",\n",
    "            \"rearrangement_file\": \"agave://data.vdjserver.org//irplus/apps/toy/mao/ls5/test/IGH_SRR1383450.airr.tsv.gz\"\n",
    "          },\n",
    "          \"parameters\": {\n",
    "            \"creator\": \"mao\",\n",
    "            \"single_flag\": true\n",
    "          }\n",
    "        }\n",
    "        test-app.json (END)    \n",
    "\n",
    "In this job we use two input files that were previously uploaded to tapis, the singularity image and the rearrangements file.\n",
    "\n",
    "Next, we will submit the test job:\n",
    "\n",
    "> tapis jobs submit -F test-app.json\n",
    "    \n",
    "outputs:\n",
    "    \n",
    "        +--------+------------------------------------------+\n",
    "        | Field  | Value                                    |\n",
    "        +--------+------------------------------------------+\n",
    "        | id     | 1ff219f4-523b-4134-8095-1d7cd7efad88-007 |\n",
    "        | name   | test_toy                                 |\n",
    "        | status | ACCEPTED                                 |\n",
    "        +--------+------------------------------------------+\n",
    " \n",
    "\n",
    "### Step 6 - List all the running/failed/finished jobs\n",
    "    \n",
    "List all your Jobs:\n",
    "\n",
    "> tapis jobs list\n",
    "    \n",
    "        +------------------------------------------+----------+----------------+\n",
    "        | id                                       | name     | status         |\n",
    "        +------------------------------------------+----------+----------------+\n",
    "        | 1ff219f4-523b-4134-8095-1d7cd7efad88-007 | test_toy | STAGING_INPUTS |\n",
    "        | b2d6b63e-08a1-4bb9-95c3-57a847637433-007 | test_toy | FINISHED       |\n",
    "        | 38e3a74e-7cdd-471c-96d1-9ec4fbf68b27-007 | test_toy | FINISHED       |\n",
    "        | 4d795099-a9a8-4f4b-82d6-4b2e32073c37-007 | test_toy | FAILED         |\n",
    "        +------------------------------------------+----------+----------------+\n",
    "\n",
    "And we can see the status of a single job\n",
    "\n",
    "> tapis jobs show 1ff219f4-523b-4134-8095-1d7cd7efad88-007\n",
    "    \n",
    "        +--------------------+------------------------------------------------------------------------------+\n",
    "        | Field              | Value                                                                        |\n",
    "        +--------------------+------------------------------------------------------------------------------+\n",
    "        | accepted           | 2020-09-03T17:48:17.508-05:00                                                |\n",
    "        | appId              | irplus-toy-ls5-mao-0.1                                                       |\n",
    "        | appUuid            | 8883588656832844266-242ac117-0001-005                                        |\n",
    "        | archive            | False                                                                        |\n",
    "        | archivePath        | mao/archive/jobs/job-1ff219f4-523b-4134-8095-1d7cd7efad88-007                |\n",
    "        | archiveSystem      | data.vdjserver.org                                                           |\n",
    "        | blockedCount       | 0                                                                            |\n",
    "        | created            | 2020-09-03T17:48:17.000-05:00                                                |\n",
    "        | ended              | None                                                                         |\n",
    "        | failedStatusChecks | 0                                                                            |\n",
    "        | id                 | 1ff219f4-523b-4134-8095-1d7cd7efad88-007                                     |\n",
    "        | lastStatusCheck    | 42 seconds ago                                                               |\n",
    "        | lastStatusMessage  | Transitioning from status QUEUED to RUNNING in phase MONITORING.             |\n",
    "        | lastUpdated        | 2020-09-03T17:50:26.000-05:00                                                |\n",
    "        | maxHours           | 1.0                                                                          |\n",
    "        | memoryPerNode      | 1.0                                                                          |\n",
    "        | name               | test_toy                                                                     |\n",
    "        | nodeCount          | 1                                                                            |\n",
    "        | owner              | mao                                                                          |\n",
    "        | processorsPerNode  | 40                                                                           |\n",
    "        | remoteEnded        | None                                                                         |\n",
    "        | remoteJobId        | 3034682                                                                      |\n",
    "        | remoteOutcome      | None                                                                         |\n",
    "        | remoteQueue        | normal                                                                       |\n",
    "        | remoteStarted      | 2020-09-03T17:50:26.801-05:00                                                |\n",
    "        | remoteStatusChecks | 1                                                                            |\n",
    "        | remoteSubmitted    | a minute ago                                                                 |\n",
    "        | schedulerJobId     | None                                                                         |\n",
    "        | status             | RUNNING                                                                      |\n",
    "        | submitRetries      | 0                                                                            |\n",
    "        | systemId           | irplus-ls5.tacc.utexas.edu                                                   |\n",
    "        | tenantId           | vdjserver.org                                                                |\n",
    "        | tenantQueue        | aloe.jobq.vdjserver.org.submit.DefaultQueue                                  |\n",
    "        | visible            | True                                                                         |\n",
    "        | workPath           | /scratch/01114/vdj/mao/job-1ff219f4-523b-4134-8095-1d7cd7efad88-007-test_toy |\n",
    "        +--------------------+------------------------------------------------------------------------------+\n",
    " \n",
    "To list the logs (stderr) from a job use:\n",
    "    \n",
    "> tapis jobs output logs 1ff219f4-523b-4134-8095-1d7cd7efad88-007\n",
    "\n",
    "        tar (child): binaries.tgz: Cannot open: No such file or directory\n",
    "\n",
    "        tar (child): Error is not recoverable: exiting now\n",
    "\n",
    "        tar: Child returned status 2\n",
    "\n",
    "        tar: Error is not recoverable: exiting now\n",
    "\n",
    "        cat: joblist: No such file or directory\n",
    "\n",
    "_The above errors are normal because Scott left some commands in the script._\n",
    "\n",
    "To list the STDOUT (standard output) of the jobs use:\n",
    "\n",
    "> tapis jobs output logs 1ff219f4-523b-4134-8095-1d7cd7efad88-007 --stdout\n",
    "\n",
    "        START at Thu Sep  3 12:50:30 CDT 2020\n",
    "\n",
    "        Input files:\n",
    "        singularity_image=immcantation_suite-4.1.0.sif\n",
    "        rearrangement_file=IGH_SRR1383450.airr.tsv.gz\n",
    "\n",
    "        Application parameters:\n",
    "        single_flag=1\n",
    "        optional_number=\n",
    "        optional_enum=\n",
    "        VERSIONS:\n",
    "        immcantation: 4.1.0\n",
    "        date: 2020.08.12\n",
    "\n",
    "        presto: 0.6.1\n",
    "        changeo: 1.0.0\n",
    "        alakazam: 1.0.2\n",
    "        shazam: 1.0.2\n",
    "        tigger: 1.0.0\n",
    "        scoper: 1.1.0\n",
    "        prestor: 0.0.5\n",
    "        rabhit: 0.1.5\n",
    "        rdi: 1.0.0\n",
    "        igphyml: 1.1.3\n",
    "        airr-py: 1.3.0\n",
    "        airr-r: 1.3.0\n",
    "        blast: 2.9.0\n",
    "        cd-hit: 4.8.1\n",
    "        igblast: 1.16.0\n",
    "        muscle: 3.8.425\n",
    "        phylip: 3.697\n",
    "        vsearch: 2.13.6\n",
    "\n",
    "        START at Thu Sep  3 12:52:07 CDT 2020\n",
    "        initProvenance\n",
    "        total 1178572\n",
    "        -rw------- 1 vdj G-803419   31140885 Sep  3 12:50 IGH_SRR1383450.airr.tsv\n",
    "        -rw------- 1 vdj G-803419 1176412160 Sep  3 12:50 immcantation_suite-4.1.0.sif\n",
    "        drwx------ 2 vdj G-803419       4096 Sep  3 12:50 test\n",
    "        -rw------- 1 vdj G-803419        229 Sep  3 12:52 test_toy-1ff219f4-523b-4134-8095-1d7cd7efad88-007.err\n",
    "        -rw------- 1 vdj G-803419        579 Sep  3 12:52 test_toy-1ff219f4-523b-4134-8095-1d7cd7efad88-007.out\n",
    "        -rwx------ 1 vdj G-803419       3147 Sep  3 12:50 test_toy.ipcexe\n",
    "        -rw------- 1 vdj G-803419       3568 Sep  3 12:50 toy.json\n",
    "        -rw------- 1 vdj G-803419       1442 Sep  3 12:50 toy.sh\n",
    "        -rw------- 1 vdj G-803419       3331 Sep  3 12:50 toy_common.sh\n",
    "        DONE at Thu Sep  3 12:52:07 CDT 2020\n",
    "\n",
    "In the end, what this toy-app does is to print the versions of the software in the incantation suite (the singularity image) and a _ls_ of the directory is it run.\n",
    "    \n",
    "### Step 7 - Retrieve outputs\n",
    "\n",
    "To download see the output files of a job we can use the _tapis jobs output_ command. We can retrieve the help with:\n",
    "\n",
    "> tapis jobs output --help\n",
    "    \n",
    "To list all the output files of a job use the job identifier:\n",
    "    \n",
    "> tapis jobs output list 1ff219f4-523b-4134-8095-1d7cd7efad88-007\n",
    "\n",
    "        +-------------------------------------------------------+---------------+------------+\n",
    "        | name                                                  | lastModified  |     length |\n",
    "        +-------------------------------------------------------+---------------+------------+\n",
    "        | IGH_SRR1383450.airr.tsv                               | 5 minutes ago |   31140885 |\n",
    "        | immcantation_suite-4.1.0.sif                          | 5 minutes ago | 1176412160 |\n",
    "        | test                                                  | 4 minutes ago |       4096 |\n",
    "        | test_toy-1ff219f4-523b-4134-8095-1d7cd7efad88-007.err | 3 minutes ago |        229 |\n",
    "        | test_toy-1ff219f4-523b-4134-8095-1d7cd7efad88-007.out | 3 minutes ago |       1293 |\n",
    "        | test_toy.ipcexe                                       | 4 minutes ago |       3147 |\n",
    "        | toy.json                                              | 4 minutes ago |       3568 |\n",
    "        | toy.sh                                                | 4 minutes ago |       1442 |\n",
    "        | toy_common.sh                                         | 4 minutes ago |       3331 |\n",
    "        +-------------------------------------------------------+---------------+------------+\n",
    "\n",
    "We can retrieve a single file from the ouput list, or all output.\n",
    "To retrieve **ALL** the output of the processing use the **Job Id** (This download takes a lot of time - be prepared):\n",
    "    \n",
    "> tapis jobs output download 4d795099-a9a8-4f4b-82d6-4b2e32073c37-007\n",
    "\n",
    "        +-------------+-------+\n",
    "        | Field       | Value |\n",
    "        +-------------+-------+\n",
    "        | downloaded  | 12    |\n",
    "        | skipped     | 0     |\n",
    "        | messages    | 0     |\n",
    "        | elapsed_sec | 616   |\n",
    "        +-------------+-------+\n",
    "\n",
    "This will create a folder with the _job id_ as a name, the downloaded contents are inside that folder.\n",
    "    \n",
    "    \n",
    "### Step 8 - Apply with other applications\n",
    "    \n",
    "Use this structure to test with other applications.  \n",
    "    If you modify any of the files, for example the shell scripts, we need to upload the files again. If any change occurs in the json files we need to use the _create_ command:\n",
    "    \n",
    "> tapis apps create -F toy.json\n",
    "    \n",
    "The downside of using the singularity image is that it occupies several Gb, if we download everything from the output of a job, we will get the singularity file also."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Requirements to wrap an App with TAPIS\n",
    "\n",
    "## Files\n",
    "\n",
    "### Minimum to create an App\n",
    "\n",
    "To create an App you need at least the Wrapper Description, the Wrapper Bash script and a Test Bash script (currently its blank).\n",
    "\n",
    "\n",
    "### Upload Script\n",
    "\n",
    "This is a simple script to upload local files into the Tapis filesystem. All input files must be in the Tapis Filesystem to be retrieved by the Tapis App Wraper.  \n",
    "The filesystem structure mirrors the github directory structure, i.e. /irplus/apps/TOOL_NAME/VERSION\n",
    "\n",
    "**NOTE**: Normally an application is compiled on the execution system, and the binaries are added to the App assets. But we will be using singularity images during the Hackathon.\n",
    "\n",
    "### App Wrapper Description (JSON)\n",
    "\n",
    "In our tutorial this corresponds to the file _toy.json_.  \n",
    "The Wrapper (description file) is divided in:\n",
    "- Important settings:\n",
    "    - name, version\n",
    "    - executionSystem (execution system name), see Step 3 above (_tapis systems list_ command)\n",
    "    - deployementPath (where the app assets are)\n",
    "    - templatePath (Wrappers Bash script)\n",
    "    - testPath (Test Bash script)\n",
    "- inputs:\n",
    "    - A list of file descriptors\n",
    "    - Files that Tapis will automatically copy to the job scratch directory\n",
    "        - e.g. data files, singularity image,\n",
    "        - minCardinality, maxCardiality, required\n",
    "- parameters:\n",
    "    - id, type, required, default, mincardinality, maxCardinality\n",
    "    - Basic JSON types, arrays and enumerations\n",
    "    - Create your own: add parameters to trigger different tool usage (e.g. if a single tool performs different unique analysis functionality)\n",
    "    \n",
    "### App Wrapper Details (Bash script)\n",
    "\n",
    "Please note that there are in fact 2 wrapper scripts:\n",
    "- common/toy_common.sh (Common along all the execution systems)\n",
    "        \n",
    "        This script contains the function that runs the Wrapped tool - run_workflow()  which is common to all the execution systems, thus not requiring to be changed or adapted as it happens with the other script below. It also shares a set of common functions that can be used within the App Wrapper script. \n",
    "        Loaded within the execution system specific wrapper script (the other one) - see around line 38 of toy.sh:\n",
    "        \n",
    "    > \\# bring in common functions  \n",
    "    > source ./toy_common.sh\n",
    "\n",
    "- ls5/toy.sh (Technically the wrapper script)\n",
    "        - assign the inputs and parameters of the App Wrapper Description (JSON) to variables.\n",
    "                NOTE: to use a input or a parameter within the script use \"${<id_value>}\",  \n",
    "                where <id_value> is the value of the id property of the object within  \n",
    "                App Wrapper description file (toy.json)\n",
    "                Example:\n",
    "                \n",
    "    > singularity_image=\"\\\\${singularity_image}\"  \n",
    "    > optional_number=\"\\\\${optional_number}\"\n",
    "                \n",
    "        - load Modules, or other pre-installed tools that are available in the HPC\n",
    "        - calls functions in the common script\n",
    "        \n",
    "Some command where left commented - for example see _laucher_ for embarrassingly-parallel multi-processing.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Singularity image\n",
    "\n",
    "See if the tool has a docker/singularity image, if not we will need to create one:\n",
    "- Setup a docker file and put the dockerfile in the tapis github\n",
    "- Get the tool running in your local docker\n",
    "- If required run a job passing the docker file using the tool singularity-ls5-0.1u1 app for building a singularity image from docker hub (not really explained how to use it).\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}