mars_craters_starting_kit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: left\">\n",
    "[<img src=\"img/logoUPSayPlusCDS_990.png\" width=\"800px\">](http://www.datascience-paris-saclay.fr)\n",
    "</div>\n",
    "\n",
    "# [RAMP](https://www.ramp.studio/problems/mars_craters) on Mars craters detection\n",
    "\n",
    "_Alexandre Boucaud (CDS), Joris van den Bossche (CDS), Balazs Kegl (CDS), Frédéric Schmidt (GEOPS), Anthony Lagain (GEOPS)_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [Introduction](#Introduction)\n",
    "2. [Preprocessing](#Preprocessing)\n",
    "3. [Workflow](#Workflow)\n",
    "4. [Evaluation](#Scoring)\n",
    "5. [Local testing/exploration](#Local-testing)\n",
    "6. [Submission](#Submitting-to-ramp.studio)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Impact craters in planetary science are used to date planetary surfaces, to characterize surface processes and to study the upper crust of terrestrial bodies in our Solar System (Melosh, 1989). Thanks to the Martian crater morphology, a wide amount of information could be deduced on the geological history of Mars, as for example the evolution of the surface erosion rate, the presence of liquid water in the past, the volcanic episodes or the volatiles layer in the subsurface (Carr & Head, 2010). These studies are widely facilitated by the availability of reference crater databases. \n",
    "\n",
    "Surveying impact craters is therefore an important task which traditionally has been achieved by means of visual inspection of images. The enormous number of craters smaller than one kilometer in diameter, present on high resolution images, makes visual counting of such craters impractical. In order to overcome this problem, several algorithms have been developed to automatically detect impact structures on planetary images (Bandeira et al., 2007 ; Martins et al., 2009). Nevertheless, these method allow to detect only 70-80 % of craters (Urbach & Stepinski, 2009)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The prediction task\n",
    "\n",
    "This challenge proposes to design the best algorithm to detect crater position and size starting from the most complete Martian crater database containing 384 584 verified impact structures larger than one kilometer of diameter (Lagain et al. 2017). We propose to give to the users a subset of this large dataset in order to test and calibrate their algorithm. \n",
    "\n",
    "<img src=\"img/craters_sample.png\" width=\"70%\">\n",
    "<div style=\"text-align: center\">Example of THEMIS reprojected data</div>\n",
    "\n",
    "\n",
    "We provide THEMIS nightime dataset (Christensen et al., 2003)\n",
    "already projected to avoid distortion, sampled at various scales and positions in form 224x224 pixels images. Using an appropriate metric, we will compare the true solution to the estimation. The goal is to provide detection of more than 90% (crater center and diameter) with a minimum number of wrong detection."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### References\n",
    "\n",
    "1. Bandeira, L.; Saraiva, J. & Pina, P., Impact Crater Recognition on Mars Based on a Probability Volume Created by Template Matching, IEEE Transactions on Geoscience and Remote Sensing, Institute of Electrical and Electronics Engineers (IEEE), 2007, 45, 4008-4015 \n",
    "\n",
    "2. Carr, M. H. and Head, J. W. III. (2010) Geologic history of Mars. Earth and Planetary Science Letters, 294, 185-203.\n",
    "\n",
    "3. Christensen, P. R.; Bandfield, J. L.; Bell, J. F.; Gorelick, N.; Hamilton, V. E.; Ivanov, A.; Jakosky, B. M.; Kieffer, H. H.; Lane, M. D.; Malin, M. C.; McConnochie, T.; McEwen, A. S.; McSween, H. Y.; Mehall, G. L.; Moersch, J. E.; Nealson, K. H.; Rice, J. W.; Richardson, M. I.; Ruff, S. W.; Smith, M. D.; Titus, T. N. & Wyatt, M. B., Morphology and Composition of the Surface of Mars: Mars Odyssey THEMIS Results, Science, 2003, 300, 2056-2061 \n",
    "\n",
    "4. Lagain, A., Marmo, C., Delaa, O., Bouley, S., Baratoux, D., Costard, F. et al. (2017) Martian crater database: 1. Reviewing and adapting to surface ages measurement. Submission to the Journal of Geophysical Research planned to November 2017.\n",
    "\n",
    "5. Martins, R.; Pina, P.; Marques, J.S & Silveira, M., Crater Detection by a Boosting Approach, IEEE Geoscience and Remote Sensing Letters, Institute of Electrical and Electronics Engineers (IEEE), 2009, 6, 127-131\n",
    "\n",
    "6. Melosh, H. J. (1989) Impact cratering: a geologic process. Oxford University Press.\n",
    "\n",
    "7. Urbach, E. and Stepinski, T. (2009) Automatic detection of sub-km craters in high resolution planetary images. Planetary and Space Science, 57, 880-887."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preprocessing\n",
    "\n",
    "### From raw data to local quadrangles\n",
    "\n",
    "The full THEMIS map of Mars has been saved in a cylindrical projection.    Raw visualization of the full map is thus quite difficult.\n",
    "\n",
    "<img src=\"img/full_image.png\" width=\"80%\">\n",
    "<div style=\"text-align: center\">Raw THEMIS data</div>\n",
    "\n",
    "In order to correct for the distortions and recover the circularity of the craters, the map has to be reprojected locally. For that, we use a partitions known as _quadrangles_. 140 quadrangles are used to cover the whole Mars surface and we have used only a few of them for this exercice.\n",
    "\n",
    "<img src=\"img/quadrangles.png\" width=\"95%\">\n",
    "<div style=\"text-align: center\">Quadrangle division in the raw THEMIS projection</div>\n",
    "\n",
    "We use the boundaries of the quadrangle to extract the relevant pixels from the raw THEMIS map (above). Here we choose the quadrangle #77, close to the equator, and thus only slightly distorted.\n",
    "\n",
    "<img src=\"img/quad_77_original_small.png\" width=\"80%\">\n",
    "<div style=\"text-align: center\">Quadrangle 77</div>\n",
    "\n",
    "Then we reproject the pixel to the local stereographic projection in order to correct for the distortions. The result can be seen below.\n",
    "\n",
    "<img src=\"img/quad_77_localstereo_small.png\" width=\"80%\">\n",
    "<div style=\"text-align: center\">Quadrangle 77 reprojected</div>\n",
    "\n",
    "For illustration, we use the crater database to project the labelled craters per category on the reprojected map.\n",
    "\n",
    "<img src=\"img/quad_77_local_craters_small.png\" width=\"80%\">\n",
    "<div style=\"text-align: center\">Quadrangle 77 reprojected with labelled craters</div>\n",
    "\n",
    "### From quadrangles to input data\n",
    "\n",
    "After selecting a common **shape for the input** of image processing models **224x224**, we start from the reprojected quadrangles and decide to create the patches from a tiling with overlap to cover the craters falling on the edges. The tiling is also performed on downsampled versions of the quadrangles, until all craters can be detected for all sizes of craters present.\n",
    "With the current set up, the **craters** contained in every image **have radii** that span the range **of 5 to 28 pixels**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With no further do, let's have a look at the data.."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The input data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download script (optional)\n",
    "\n",
    "If the data has not yet been downloaded locally, uncomment the following cell and run it.\n",
    "\n",
    "There are ~800Mb of images."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# !python download_data.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Load the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from __future__ import division, print_function\n",
    "\n",
    "import os\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import rampwf as rw\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "from matplotlib import cm\n",
    "from matplotlib.patches import Circle\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = np.load('data/data_train.npy', mmap_mode='r')\n",
    "\n",
    "print(\"The train data is made of {} images of {}x{} pixels\".format(*X_train.shape))\n",
    "\n",
    "n_img = len(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we display 3 randomly select images, you can play around with the values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "idx_list = [30, 3470, 9030]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(ncols=3, figsize=(15, 5))\n",
    "for i, idx in enumerate(idx_list):\n",
    "    axes[i].imshow(X_train[idx], cmap='Greys_r')\n",
    "    axes[i].set_title('%d' % idx)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most images have some craters but the labeled craters are only the one whose diameter is superior to 10 pixels."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train = pd.read_csv('data/labels_train.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The labels consist of a `pandas.DataFrame` containing the list of craters. For each craters, the columns are\n",
    "- `id`: index of the image it belongs to\n",
    "- `row_p`, `col_p` : pixel position\n",
    "- `radius_p` : pixel radius\n",
    "\n",
    "Let's visualize some."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(15, 10))\n",
    "for i, idx in enumerate(idx_list):\n",
    "    img = X_train[idx]\n",
    "    lab = y_train[y_train['i'] == idx][['row_p', 'col_p', 'radius_p']].values\n",
    "    # First row images only\n",
    "    axes[0, i].imshow(img, cmap='Greys_r')\n",
    "    axes[0, i].set_title('%d' % idx)\n",
    "    # Second row, labels overlaid on the image\n",
    "    axes[1, i].imshow(img, cmap='Greys_r')\n",
    "    if lab.size != 0:\n",
    "        for y, x, radius in lab:\n",
    "            crater = Circle((x, y), radius, color='r', ec=None, alpha=0.5)\n",
    "            axes[1, i].add_patch(crater)\n",
    "fig.tight_layout()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Labels distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Individual craters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "print(\"There are {} labeled craters in the training set\".format(y_train.size))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lots of \"empty\" images"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "n_img_withcraters = y_train['i'].unique().size\n",
    "\n",
    "print(\"There are labeled craters for only {:.1%} of the images\".format(n_img_withcraters / n_img))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_train.radius_p.plot(kind='hist', bins=20, log=True)\n",
    "plt.xlabel('Radius of craters [pixels]')\n",
    "plt.title('Crater radius frequency for the training data')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/workflow.svg\" width=\"70%\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For submitting at the [RAMP site](http://ramp.studio), you will have to write one class, saved in a specific file:   \n",
    "\n",
    "* a class `ObjectDetector` that will handle the training and prediction in a `object_detector.py` file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Object detector\n",
    "\n",
    "A dummy object detector which detects a single crater in the middle of each "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "%load submissions/starting_kit/object_detector.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Format of the data it receives:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X, y = problem.get_train_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A single image patch:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X[0].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The corresponding ground thruth data for that patch consists of a list of tuples (x, y, radius):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y[2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y[4]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The format returned by the `predict` method contains one extra element: the confidence or score for that predicted crater: `(conf, x, y, radius)`. The returned value in `predict` should then be an array of lists of tuples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "!ramp_test_submission --quick-test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scoring\n",
    "\n",
    "`y_true` and `y_pred` can have variable length\n",
    "\n",
    "- What is a detection ?\n",
    "- What is the distance between two lists of predictions ?\n",
    "- How do we rank the various models ?\n",
    "\n",
    "\n",
    "#### Intersetion over Union : IoU\n",
    "\n",
    "Distance in (x, y, r) space computed using IoU (adapted for circles in this problem)\n",
    "\n",
    "<img src=\"img/iou_formula.png\" width=\"30%\">\n",
    "<img src=\"img/iou_examples.png\" width=\"70%\">\n",
    "\n",
    "Detection can start at `IoU > 0` but traditionally an object is detected if **`IoU > 0.5`**.\n",
    "\n",
    "#### Confidence level\n",
    "\n",
    "Each prediction should be associated with a confidence, so that they can be ranked. The confidence value should be the first one that appears, before `x, y, radius`.\n",
    "\n",
    "#### Matching\n",
    "\n",
    "The Hungarian method [`O(N log(N))`] is used to compute the best match between the true and predicted list.\n",
    "\n",
    "### Metric #1 Average Precision\n",
    "\n",
    "- **precision**: fraction of correct predictions\n",
    "- **recall**: fraction of true objects that are predicted\n",
    "\n",
    "Precision and recall are computed as a function of the confidence value threshold to produce the mean Average Precision (mAP) curve.\n",
    "\n",
    "<img src=\"img/precision-recall-curve.png\" width=\"70%\">\n",
    "\n",
    "\n",
    "### Metric #2 OSPA\n",
    "\n",
    "[OSPA](http://www.dominic.schuhmacher.name/papers/ospa.pdf) is a miss-distance between the best match of two lists of objects with various size. It penalizes the both the distance between the best match entries and the size difference, referred to as _cardinality_.\n",
    "\n",
    "The distance used here is the IoU for circles.\n",
    "\n",
    "<img src=\"img/ospa_figure2.png\" width=\"80%\">\n",
    "<div style=\"text-align: center\">Various examples of sets of entries and their corresponding OSPA value.</div>\n",
    "\n",
    "### Metric #3 SCP\n",
    "\n",
    "_Superposed Cylinders and Prisms_ is a new score based on the projection of true and predicted craters on a rasterized image (mask).\n",
    "\n",
    "We construct the mask by projecting circles: true craters as positive and predicted ones as negative.\n",
    "\n",
    "<img src=\"img/det_mask.png\" width=\"65%\">\n",
    "\n",
    "The craters are **inverse weighted** by their area so the sum over a given crater is normalized to unity.\n",
    "\n",
    "The **score** is obtained as the sum of the absolute value of the mask."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some attention points when building the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-danger\">\n",
    "\n",
    " <ul>\n",
    "  <li>In the ground truth labels, **only craters are present with a radius of 5 to 28 pixels**. You should make sure your model also only predicts craters in this range, otherwise it will be penalized for finding (possibly existing) craters not present in the ground truth.</li>\n",
    "  <li>All metrics, except for Average Precision, are calculated by selecting the set of predictions based on a **confidence threshold of 0.5**. So you should **rescale the returned value of the confidence** so that the best threshold for your model is rescaled to 0.5.</li>\n",
    "  <li>The data for training and testing comes from different quadrangles of Mars, and can different characteristics (different lightning angle, different image quality, ..). This means it will be important to make the model robust to that (e.g. by randomly flipping the images).</li>\n",
    "</ul> \n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Image processing example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep net example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Deep learning for object detection**\n",
    "\n",
    "Different approaches and published models:\n",
    "\n",
    "- Fast(er) R-CNN (https://arxiv.org/abs/1506.01497): region proposal\n",
    "- YOLO (You Only Look Once, [paper](https://www.cv-foundation.org/openaccess/content_cvpr_2016/papers/Redmon_You_Only_Look_CVPR_2016_paper.pdf))\n",
    "- SSD (Single Shot MultiBox Detector, https://arxiv.org/abs/1512.02325)\n",
    "- ...\n",
    "\n",
    "The starting kit contains a `keras_ssd7` example, using a simplied version of SSD and using the implementation of https://github.com/pierluigiferrari/ssd_keras/ (there are implementations as well: https://github.com/rykov8/ssd_keras/, https://github.com/amdegroot/ssd.pytorch)\n",
    "\n",
    "Some characteristics of the Single Shot MultiBox Detector (SSD):\n",
    "\n",
    "- Set of default boxes, predict offset and confidence\n",
    "- Multi-scale feature maps for detection\n",
    "- Default boxes with different aspect ratios\n",
    "\n",
    "\n",
    "<img src=\"img/ssd_architecture.png\">\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The use the starting kit version, you need to install the following package:"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "pip install git+https://github.com/paris-saclay-cds/ssd_keras.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's run the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import problem"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train, y_train = problem.get_train_data()\n",
    "X_test, y_test = problem.get_train_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('./submissions/keras_ssd7/')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from object_detector import ObjectDetector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model = ObjectDetector()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training the model with `fit` (in the starting kit example, we load some pretrained weights to be able to show results here, but note this is not possible for an actual submission):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Predict part of the test data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "y_pred = model.predict(X_test[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred[0].shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred[0][0].tolist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the predictions and some scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Draw the predicted boxes onto the image\n",
    "\n",
    "i = 0\n",
    "\n",
    "fig, ax = plt.subplots(figsize=(20,12))\n",
    "ax.imshow(X_test[0], cmap=plt.cm.gray)\n",
    "\n",
    "# Draw the predicted boxes in red\n",
    "for conf, x, y, r in y_pred[i][-5:]:\n",
    "    ax.add_patch(plt.Rectangle((y - r, x - r), r*2, r*2, color='red', fill=False, linewidth=2))  \n",
    "\n",
    "# Draw the ground truth boxes in green (omit the label for more clarity)\n",
    "for x, y, r in y_test[i]:\n",
    "    ax.add_patch(plt.Rectangle((y-r, x-r), r*2, r*2, color='green', fill=False, linewidth=2))  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import rampwf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from rampwf.score_types.detection import AverageDetectionPrecision, DetectionPrecision, DetectionRecall, SCP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "average_precision = AverageDetectionPrecision()\n",
    "precision = DetectionPrecision()\n",
    "recall = DetectionRecall()\n",
    "scp = SCP(shape=(224, 224), minipatch=[56, 168, 56, 168])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ap = average_precision(y_test[:500], y_pred)\n",
    "ap"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "precision(y_test[:500], y_pred)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "recall(y_test[:500], y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Those number depend on the confidence threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ps = []\n",
    "rs = []\n",
    "scps = []\n",
    "\n",
    "for conf_threshold in np.linspace(0, 1, 50):\n",
    "    ps.append(precision(y_test[:100], y_pred, conf_threshold=conf_threshold))\n",
    "    rs.append(recall(y_test[:100], y_pred, conf_threshold=conf_threshold))\n",
    "    scps.append(scp(y_test[:100], y_pred, conf_threshold=conf_threshold))\n",
    "    \n",
    "ps = np.array(ps)\n",
    "rs = np.array(rs)\n",
    "scps = np.array(scps)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Precision-recall curve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.plot(rs, ps, 'o-')\n",
    "ax.set_xlabel('Recall', fontsize=16)\n",
    "ax.set_ylabel('Precision', fontsize=16)\n",
    "ax.set(xlim=(0, 1), ylim=(0, 1))\n",
    "ax.text(0.7, 0.8, 'AP = {:.2f}'.format(ap), fontsize=16)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "SCP, precision and recall in function of the confidence threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "conf_thresholds = np.linspace(0, 1, 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "ax.plot(conf_thresholds, scps, label='scp')\n",
    "ax.plot(conf_thresholds, ps, 'C1', label='precision')\n",
    "ax.plot(conf_thresholds, rs, 'C2', label='recall')\n",
    "ax.plot(conf_thresholds, 1 - (2*(ps * rs) / (ps + rs)) , 'C3', label='f1 score')\n",
    "\n",
    "ax.legend(loc=7)\n",
    "\n",
    "ax.set_xlabel(\"Confidence threshold\")\n",
    "ax.set_ylabel(\"Score\")\n",
    "\n",
    "ax.set_ylim(0, 1)\n",
    "\n",
    "#ax.axhline(1, linestyle='--', color='grey')\n",
    "#ax.axvline(conf_thresholds[17], color='grey', linestyle='--')\n",
    "ax.spines['top'].set(linestyle='--', color='grey')\n",
    "ax.spines['right'].set(linestyle='--', color='grey')\n",
    "    \n",
    "ax.text(0.8, 0.87, 'AP = {:.2f}'.format(ap))\n",
    "ax.text(0.7, 0.80, 'min(MD) = {:.2f}'.format(np.min(scps)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Local testing/exploration\n",
    "\n",
    "It is <b><span style=\"color:red\">important that you test your submission files before submitting them</span></b>. For this we provide a unit test. Note that the test runs on your files in [`submissions/starting_kit`](/tree/submissions/starting_kit), not on the classes defined in the cells of this notebook.\n",
    "\n",
    "Check list\n",
    "\n",
    "- Install the `ramp-workflow` locally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!pip install git+https://github.com/paris-saclay-cds/ramp-workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Make sure that the python files `object_detector.py` is in the  [`submissions/starting_kit`](/tree/submissions/starting_kit) folder, and the train and test data are in [`data`](/tree/data)\n",
    "- If you haven't yet, downlad the images by executing \n",
    "  ```\n",
    "  python download_data.py\n",
    "  ```\n",
    "\n",
    "Finally, make sure the local processing goes through by running the\n",
    "```\n",
    "ramp_test_submission\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "!ramp_test_submission"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you get to see the train and test scores, and no errors, then you can submit your model to the ramp.studio."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submitting to [ramp.studio](http://ramp.studio)\n",
    "\n",
    "Once you found a good model, you can submit them to [ramp.studio](http://www.ramp.studio). First, if it is your first time using RAMP, [sign up](http://www.ramp.studio/sign_up), otherwise [log in](http://www.ramp.studio/login). Then find an open event on the particular problem, for example, the event [MNIST](http://www.ramp.studio/events/MNIST) for this RAMP. Sign up for the event. Both signups are controled by RAMP administrators, so there **can be a delay between asking for signup and being able to submit**.\n",
    "\n",
    "Once your signup request is accepted, you can go to your [sandbox](http://www.ramp.studio/events/MNIST/sandbox) and copy-paste (or upload) [`image_preprocessor.py`](/edit/submissions/starting_kit/image_preprocessor.py) and [`batch_classifier.py`](/edit/submissions/starting_kit/batch_classifier.py) from `submissions/starting_kit`. Save it, rename it, then submit it. The submission is trained and tested on our backend in the same way as `ramp_test_submission` does it locally. While your submission is waiting in the queue and being trained, you can find it in the \"New submissions (pending training)\" table in [my submissions](http://www.ramp.studio/events/MNIST/my_submissions). Once it is trained, you get a mail, and your submission shows up on the [public leaderboard](http://www.ramp.studio/events/MNIST/leaderboard). \n",
    "If there is an error (despite having tested your submission locally with `ramp_test_submission`), it will show up in the \"Failed submissions\" table in [my submissions](http://www.ramp.studio/events/MNIST/my_submissions). You can click on the error to see part of the trace.\n",
    "\n",
    "After submission, do not forget to give credits to the previous submissions you reused or integrated into your submission.\n",
    "\n",
    "The data set we use at the backend is usually different from what you find in the starting kit, so the score may be different.\n",
    "\n",
    "The usual way to work with RAMP is to explore solutions, add feature transformations, select models, perhaps do some AutoML/hyperopt, etc., _locally_, and checking them with `ramp_test_submission`. The script prints mean cross-validation scores \n",
    "```\n",
    "----------------------------\n",
    "train acc = 0.55 ± 0.0\n",
    "train nll = 14.53 ± 0.0\n",
    "valid acc = 0.55 ± 0.0\n",
    "valid nll = 14.65 ± 0.0\n",
    "test acc = 0.55 ± 0.0\n",
    "test nll = 14.6 ± 0.0\n",
    "```\n",
    "The official score in this RAMP (the first score column after \"historical contributivity\" on the [leaderboard](http://www.ramp.studio/events/MNIST/leaderboard)) is balanced accuracy aka macro-averaged recall, so the line that is relevant in the output of `ramp_test_submission` is `valid acc = 0.55 ± 0.0`. When the score is good enough, you can submit it at the RAMP."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More information\n",
    "\n",
    "You can find more information in the [README](https://github.com/paris-saclay-cds/ramp-workflow/blob/master/README.md) of the [ramp-workflow library](https://github.com/paris-saclay-cds/ramp-workflow)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contact\n",
    "\n",
    "Don't hesitate to [contact us](mailto:admin@ramp.studio?subject=Mars crater notebook)."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  },
  "nav_menu": {},
  "toc": {
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 6,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}