python/notebook/examples/processing_cli.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing Examples via CLI\n",
    "\n",
    "This effectively does the sample thing as the [processing_py.ipynb](processing_py.ipynb) notebook and you should look at that example first since it explains what many of the arguments mean in a more detail.  This version is more abbreviated, but is helpful if/when running processing through python is less convenient than a CLI."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CLI Usage\n",
    "\n",
    "This is the top level CLI app, and you can see the commands available like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type:        Celldom\n",
      "String form: <__main__.Celldom object at 0x7fd3fb86ccf8>\n",
      "File:        /usr/local/bin/celldom\n",
      "Docstring:   Celldom CLI Application\n",
      "\n",
      "Usage:       celldom \n",
      "             celldom analyze-growth-rates\n",
      "             celldom run-processor\n"
     ]
    }
   ],
   "source": [
    "!celldom"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"run-processor\" function is what will be used here which has the usage:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Type:        method\n",
      "String form: <bound method Celldom.run_processor of <__main__.Celldom object at 0x7f3123382438>>\n",
      "File:        /usr/local/bin/celldom\n",
      "Line:        20\n",
      "Docstring:   Run cell counting/cytometry for a given experiment configuration and set of raw data files\n",
      "\n",
      "Args:\n",
      "    experiment_config_path: Path to experiment configuration\n",
      "        (e.g. /lab/repos/celldom/config/experiment/experiment_example_01.yaml)\n",
      "    data_file_patterns: Input image path glob patterns as either a single string or list of strings; Examples:\n",
      "        - \"/lab/data/dataset/dataset03/*/*.tif\"\n",
      "        - [\"/lab/data/dataset/dataset03/*Chip1/*.tif\",\"/lab/data/dataset/dataset03/*Chip3/*.tif\"]\n",
      "    output_dir: Directory in which results will be stored\n",
      "    sample_rate: Float in (0, 1] indicating a fractional sampling rate of raw files to use\n",
      "    sample_count: Fixed number of raw files to limit processing to\n",
      "\n",
      "Usage:       celldom run-processor EXPERIMENT_CONFIG_PATH DATA_FILE_PATTERNS OUTPUT_DIR [SAMPLE_RATE] [SAMPLE_COUNT]\n",
      "             celldom run-processor --experiment-config-path EXPERIMENT_CONFIG_PATH --data-file-patterns DATA_FILE_PATTERNS --output-dir OUTPUT_DIR [--sample-rate SAMPLE_RATE] [--sample-count SAMPLE_COUNT]\n"
     ]
    }
   ],
   "source": [
    "!celldom run-processor -- --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Execution\n",
    "\n",
    "**Note**: All of this should usually be done at a command line but it's done in a notebook here just so it's easier to share/save as an example\n",
    "\n",
    "This example will process 10 randomly selected files:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:cli:Found 31759 raw data files to process\n",
      "INFO:cli:Randomly selecting 10 files to process\n",
      "INFO:cli:Number of raw data files after sample: 10\n",
      "INFO:cli:Loading experiment configuration from path: /lab/repos/celldom/config/experiment/experiment_example_01.yaml\n",
      "INFO:cli:Running data processor (output dir = /lab/data/celldom/output/experiment_00) ...\n",
      "Using TensorFlow backend.\n",
      "100%|###########################################| 10/10 [00:44<00:00,  4.45s/it]\n",
      "INFO:cli:Processing complete\n"
     ]
    }
   ],
   "source": [
    "!celldom run-processor \\\n",
    "--experiment-config-path=/lab/repos/celldom/config/experiment/experiment_example_01.yaml \\\n",
    "--data-file-patterns=[\"'/lab/data/celldom/dataset/dataset03/*MOLM13*Chip2/*.tif'\"] \\\n",
    "--output-dir=/lab/data/celldom/output/experiment_00 \\\n",
    "--sample-count=10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/lab/data/celldom/output/experiment_00/data.h5\n"
     ]
    }
   ],
   "source": [
    "# All results have now been written to here:\n",
    "!find /lab/data/celldom/output/experiment_00/*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 2385 entries, 0 to 3\n",
      "Data columns (total 22 columns):\n",
      "acq_apartment        2385 non-null object\n",
      "acq_chip             2385 non-null object\n",
      "acq_cohort           2385 non-null object\n",
      "acq_concentration    2385 non-null object\n",
      "acq_datetime         2385 non-null datetime64[ns]\n",
      "acq_id               2385 non-null object\n",
      "acq_processed_at     2385 non-null datetime64[ns]\n",
      "acq_street           2385 non-null object\n",
      "acq_z                2385 non-null object\n",
      "apt_id               2385 non-null int64\n",
      "apt_num              2385 non-null object\n",
      "area                 2385 non-null int64\n",
      "centroid_x           2385 non-null float64\n",
      "centroid_y           2385 non-null float64\n",
      "eccentricity         2385 non-null float64\n",
      "roi_xmax             2385 non-null int64\n",
      "roi_xmin             2385 non-null int64\n",
      "roi_ymax             2385 non-null int64\n",
      "roi_ymin             2385 non-null int64\n",
      "score                2385 non-null float64\n",
      "solidity             2385 non-null float64\n",
      "st_num               2385 non-null object\n",
      "dtypes: datetime64[ns](2), float64(5), int64(6), object(9)\n",
      "memory usage: 428.6+ KB\n"
     ]
    }
   ],
   "source": [
    "# And you could explore this more in python like:\n",
    "import pandas as pd\n",
    "store = pd.HDFStore('/lab/data/celldom/output/experiment_00/data.h5', 'r')\n",
    "store.get('table_cell').info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analysis\n",
    "\n",
    "Following processing, there are commands like this as well to do some basic analysis.  Results from these are usually notebooks, and the paths to those notebooks is printed when the command is done (which you can open in jupyterlab):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:root:Generating grammar tables from /usr/lib/python3.5/lib2to3/Grammar.txt\n",
      "INFO:root:Generating grammar tables from /usr/lib/python3.5/lib2to3/PatternGrammar.txt\n",
      "INFO:cli:Executing notebook \"/lab/repos/celldom/python/notebook/templates/growth_rate_analysis_01.ipynb\"\n",
      "Input Notebook:  /lab/repos/celldom/python/notebook/templates/growth_rate_analysis_01.ipynb\n",
      "Output Notebook: /lab/data/celldom/output/experiment_01/notebook/growth_rate_analysis_01.ipynb\n",
      "INFO:traitlets:Executing notebook with kernel: python3\n",
      "100%|###########################################| 27/27 [00:32<00:00,  1.21s/it]\n",
      "INFO:cli:Analysis complete; results saved to /lab/data/celldom/output/experiment_01/notebook/growth_rate_analysis_01.ipynb`\n"
     ]
    }
   ],
   "source": [
    "!celldom analyze_growth_rates --data-dir=/lab/data/celldom/output/experiment_01"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}