2 - Solvers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# Global optimization solvers"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 1. Basin Hopping"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1.1 Description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "This is a Dashboard interface for the <a href=\"https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.optimize.basinhopping.html\">Basin hopping</a> from the scipy package. \n",
    "\n",
    "Basin-hopping is a stochastic algorithm which attempts to find the global minimum of a smooth scalar function of one or more variables [R144] [R145] [R146] [R147]. The algorithm in its current form was described by David Wales and Jonathan Doye [R145] http://www-wales.ch.cam.ac.uk/.\n",
    "\n",
    "The algorithm is iterative with each cycle composed of the following features\n",
    "\n",
    "    random perturbation of the coordinates\n",
    "    local minimization\n",
    "    accept or reject the new coordinates based on the minimized function value\n",
    "\n",
    "The acceptance test used here is the Metropolis criterion of standard Monte Carlo algorithms, although there are many other possibilities [R146].\n",
    "\n",
    "This is a graphical description of how it works:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"solvers_data/bh.png\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "## 1.2 Implementation and use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "import time#\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import copy\n",
    "\n",
    "from scipy.optimize import  differential_evolution,minimize\n",
    "from scipy.optimize import basinhopping\n",
    "from scipy.optimize import  differential_evolution\n",
    "\n",
    "from IPython.core.display import clear_output\n",
    "\n",
    "from shaolin.core.dashboard import Dashboard\n",
    "from test_functions import TestFunctions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class MyBounds(object):\n",
    "    \"\"\"Class in charge of managing the boundaries of the sampling region for the basin hopping solver.\"\"\"\n",
    "    def __init__(self, xmax=[1.1,1.1], xmin=[-1.1,-1.1],other_test=None ):\n",
    "        def true(**kwargs):\n",
    "            return True\n",
    "        self._outer_accept_test = true if other_test is None else other_test\n",
    "        self.xmax = np.array(xmax)\n",
    "        self.xmin = np.array(xmin)\n",
    "\n",
    "    def __call__(self, **kwargs):\n",
    "        x = kwargs[\"x_new\"]\n",
    "        out = self._outer_accept_test(**kwargs)\n",
    "        tmax = bool(np.all(x <= self.xmax))\n",
    "        tmin = bool(np.all(x >= self.xmin))\n",
    "        return tmax and tmin and out\n",
    "\n",
    "class BasinHopping(Dashboard):\n",
    "    \"\"\"Find the global minimum of a function using the basin-hopping algorithm\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    objfunc: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    x0 : ndarray\n",
    "        Initial guess.\n",
    "    niter : integer, optional\n",
    "        The number of basin hopping iterations\n",
    "    T : float, optional\n",
    "        The \"temperature\" parameter for the accept or reject criterion.  Higher\n",
    "        \"temperatures\" mean that larger jumps in function value will be\n",
    "        accepted.  For best results ``T`` should be comparable to the\n",
    "        separation\n",
    "        (in function value) between local minima.\n",
    "    stepsize : float, optional\n",
    "        initial step size for use in the random displacement.\n",
    "    minimizer_kwargs : dict, optional\n",
    "        Extra keyword arguments to be passed to the minimizer\n",
    "        ``scipy.optimize.minimize()`` Some important options could be:\n",
    "            method : str\n",
    "                The minimization method (e.g. ``\"L-BFGS-B\"``)\n",
    "            args : tuple\n",
    "                Extra arguments passed to the objective function (``func``) and\n",
    "                its derivatives (Jacobian, Hessian).\n",
    "    take_step : callable ``take_step(x)``, optional\n",
    "        Replace the default step taking routine with this routine.  The default\n",
    "        step taking routine is a random displacement of the coordinates, but\n",
    "        other step taking algorithms may be better for some systems.\n",
    "        ``take_step`` can optionally have the attribute ``take_step.stepsize``.\n",
    "        If this attribute exists, then ``basinhopping`` will adjust\n",
    "        ``take_step.stepsize`` in order to try to optimize the global minimum\n",
    "        search.\n",
    "    accept_test : callable, ``accept_test(f_new=f_new, x_new=x_new, f_old=fold, x_old=x_old)``, optional\n",
    "        Define a test which will be used to judge whether or not to accept the\n",
    "        step.  This will be used in addition to the Metropolis test based on\n",
    "        \"temperature\" ``T``.  The acceptable return values are True,\n",
    "        False, or ``\"force accept\"``. If any of the tests return False\n",
    "        then the step is rejected. If the latter, then this will override any\n",
    "        other tests in order to accept the step. This can be used, for example,\n",
    "        to forcefully escape from a local minimum that ``basinhopping`` is\n",
    "        trapped in.\n",
    "    callback : callable, ``callback(x, f, accept)``, optional\n",
    "        A callback function which will be called for all minima found.  ``x``\n",
    "        and ``f`` are the coordinates and function value of the trial minimum,\n",
    "        and ``accept`` is whether or not that minimum was accepted.  This can be\n",
    "        used, for example, to save the lowest N minima found.  Also,\n",
    "        ``callback`` can be used to specify a user defined stop criterion by\n",
    "        optionally returning True to stop the ``basinhopping`` routine.\n",
    "    interval : integer, optional\n",
    "        interval for how often to update the ``stepsize``\n",
    "    disp : bool, optional\n",
    "        Set to True to print status messages\n",
    "    niter_success : integer, optional\n",
    "        Stop the run if the global minimum candidate remains the same for this\n",
    "        number of iterations.\n",
    "        \n",
    "        \n",
    "    Attributes\n",
    "    ----------\n",
    "    epoch: int\n",
    "        Current iteration of the selected solver.\n",
    "    pot_func: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    best: (aray,float),  (n_components,),1\n",
    "        Tuple containing the minimum value found for the objective function\n",
    "        and the parameters that yielded that value.\n",
    "    best_pos_so_far: (array), (n_components)\n",
    "        Alias for the best parameters found by the optimizer.\n",
    "    best_so_far: float\n",
    "        Alias for minimum value found by the optimizer.\n",
    "        \"\"\"\n",
    "    def __init__(self,\n",
    "                 objfunc,\n",
    "                 x0=None,\n",
    "                 max_reads=10000000,\n",
    "                 niter=100,\n",
    "                 T=1.0,\n",
    "                 stepsize=0.5,\n",
    "                 minimizer_kwargs=None,\n",
    "                 take_step=None,\n",
    "                 accept_test=None,\n",
    "                 callback=None,\n",
    "                 interval=50,\n",
    "                 disp=False,\n",
    "                 niter_success=None,\n",
    "                 **kwargs):\n",
    "        def false(**kwargs):\n",
    "            return False\n",
    "        def true(**kwargs):\n",
    "            return True\n",
    "        self._outer_accept_test = true if accept_test is None else accept_test\n",
    "        self.minimizer_kwargs = minimizer_kwargs\n",
    "        self.take_step = take_step\n",
    "        \n",
    "       \n",
    "        self.pot_func = objfunc\n",
    "        self.x0 = x0\n",
    "        self.best_pos_so_far = None\n",
    "        self.best_so_far = 1e20\n",
    "        self._ext_callback =lambda x,val,accept: False if callback is None else callback\n",
    "        self._run_callback = false\n",
    "        self.epoch=0\n",
    "        niter_success = 0 if niter_success is None else niter_success\n",
    "        strats = ['best1bin','best1exp','rand1exp','randtobest1exp','best2exp','rand2exp','randtobest1bin','best2bin','rand2bin','rand1bin']\n",
    "        dash = ['r$N=differential_evolution',[\"##Basin-hopping$n=title\",\n",
    "                                              ['r$N=controls_row',[[\"c$N=first_row\",[\"(0,100000,1,\"+str(niter)+\")$d=Niter&n=max_epoch\",\n",
    "                                                               \"(0,1e8,1,\"+str(max_reads)+\")$d=Max reads\",\n",
    "                                                               \"(0,1e4,1,\"+str(int(niter_success))+\")$d=niter_success\"]],\n",
    "                                             \n",
    "                                              [\"c$N=sec_row\",[\"(0.,10000.,.01,\"+str(T)+\")$d=T\",\n",
    "                                                              \"(0,1e8,1,\"+str(interval)+\")$d=Interval\",\n",
    "                                                               \"(0.,100.,0.01,\"+str(stepsize)+\")$d=Stepsize\"]\n",
    "                                              ],\n",
    "                                              [\"c$n=btn_col\",[\"[\"+str(bool(disp))+\"]$d=Disp\",\"btn$d=Run&n=run_btn\"]]]\n",
    "               ]]]\n",
    "        Dashboard.__init__(self,dash,**kwargs)\n",
    "        self.run_btn.observe(self.run)\n",
    "    \n",
    "    \n",
    "    def accept_test(self,**kwargs):\n",
    "        \"\"\"Custom acceptance test support\"\"\"\n",
    "        out = self._outer_accept_test(**kwargs)\n",
    "        return out or self.pot_func.in_domain(kwargs['x'])\n",
    "    \n",
    "    \n",
    "    \n",
    "    @property\n",
    "    def best(self):\n",
    "        return (self.best_pos_so_far.copy(), float(self.best_so_far))\n",
    "    \n",
    "        \n",
    "    \n",
    "    def run(self,_=None, end_callback=None):\n",
    "        \"\"\"Solves the objective function\"\"\"\n",
    "        self.reset()\n",
    "        if not end_callback is None:\n",
    "            self._run_callback = end_callback\n",
    "        niter_success = None if self.niter_success.value == 0 else int(self.niter_success.value)\n",
    "        x0 = self.pot_func.random_in_domain()\n",
    "        \n",
    "        xmax = [x[1] for x in self.pot_func.domain]\n",
    "        xmin = [x[0] for x in self.pot_func.domain]\n",
    "        \n",
    "        bounds = MyBounds(xmax=xmax,xmin=xmin,other_test=self._outer_accept_test)\n",
    "        result = basinhopping(self.pot_func.evaluate,\n",
    "                              x0,\n",
    "                              niter=self.max_epoch.value,\n",
    "                              T=self.t.value,\n",
    "                              stepsize=self.stepsize.value,\n",
    "                              minimizer_kwargs=self.minimizer_kwargs,\n",
    "                              take_step=self.take_step,\n",
    "                              accept_test=bounds,\n",
    "                              callback=self.callback,\n",
    "                              interval=self.interval.value,\n",
    "                              disp=self.disp.value,\n",
    "                              niter_success=niter_success)\n",
    "        \n",
    "        clear_output(True)\n",
    "        print(result)\n",
    "        self.update_best(result.x)\n",
    "\n",
    "    def callback(self,x,val,accept):\n",
    "        \"\"\"custom callback for finishing the optimization in a scipy-like manner.\"\"\"\n",
    "        self.update_best(x)\n",
    "        self.epoch += 1\n",
    "        end = self._ext_callback(x,val,accept)\n",
    "        return end or self._run_callback() \n",
    "    \n",
    "    def reset(self):\n",
    "        \"\"\"Resets the solver attributes\"\"\"\n",
    "        self.best_pos_so_far = None\n",
    "        self.best_so_far = 1e20\n",
    "        self.epoch=0\n",
    "        self.pot_func.n_reads = 0\n",
    "    \n",
    "    def update_best(self,x):\n",
    "        \"\"\"Keeps track of the best value and parameters found by the solver\"\"\"\n",
    "        val = self.pot_func.evaluate(x)\n",
    "        if self.best_so_far>val:\n",
    "            self.best_pos_so_far = x\n",
    "            self.best_so_far = val"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "First, let's select the Levy's function nº13 from our TestFunctions class:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "funcs = TestFunctions()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "levy13 = funcs.functions['levy13']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "Now we can pass it as a parameter for the Dashboard."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "bh = BasinHopping(levy13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        fun: 1.921671616497636e-16\n",
      " lowest_optimization_result:       fun: 1.921671616497636e-16\n",
      " hess_inv: array([[  5.58520193e-03,  -7.67311429e-06],\n",
      "       [ -7.67311429e-06,   4.99999044e-01]])\n",
      "      jac: array([  1.15770956e-06,  -5.21661010e-09])\n",
      "  message: 'Optimization terminated successfully.'\n",
      "     nfev: 40\n",
      "      nit: 4\n",
      "     njev: 10\n",
      "   status: 0\n",
      "  success: True\n",
      "        x: array([ 1.        ,  0.99999999])\n",
      "                    message: ['requested number of basinhopping iterations completed successfully']\n",
      "      minimization_failures: 0\n",
      "                       nfev: 5456\n",
      "                        nit: 100\n",
      "                       njev: 1364\n",
      "                          x: array([ 1.        ,  0.99999999])\n"
     ]
    }
   ],
   "source": [
    "bh.widget"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"solvers_data/bh_widget.png\"></img>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "You can selct the desired parameters for the algorithm using the widgets, and once you are ready click Run to start the optimization process. Once it has finished the best value found will be displayed. In order to access it you can also access the *best* attribute o the class. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 1.        ,  0.99999999]), 1.921671616497636e-16)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bh.best"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 2. Differential evolution"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<a href=\"https://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.optimize.differential_evolution.html\">Differential evolution</a> is a stochastic population based method that is useful for global optimization problems. At each pass through the population the algorithm mutates each candidate solution by mixing with other candidate solutions to create a trial candidate. There are several strategies [R151] for creating trial candidates, which suit some problems more than others. The ‘best1bin’ strategy is a good starting point for many systems. In this strategy two members of the population are randomly chosen. Their difference is used to mutate the best member (the best in best1bin), b0, so far:\n",
    "\n",
    "**b′=b0+mutation∗(population[rand0]−population[rand1])**\n",
    "\n",
    "A trial vector is then constructed. Starting with a randomly chosen ‘i’th parameter the trial is sequentially filled (in modulo) with parameters from b’ or the original candidate. The choice of whether to use b’ or the original candidate is made with a binomial distribution (the ‘bin’ in ‘best1bin’) - a random number in [0, 1) is generated. If this number is less than the recombination constant then the parameter is loaded from b’, otherwise it is loaded from the original candidate. The final parameter is always loaded from b’. Once the trial candidate is built its fitness is assessed. If the trial is better than the original candidate then it takes its place. If it is also better than the best overall candidate it also replaces that. To improve your chances of finding a global minimum use higher popsize values, with higher mutation and (dithering), but lower recombination values. This has the effect of widening the search radius, but slowing convergence."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class DifferentialEvolution(Dashboard):\n",
    "    \"\"\"Shaolin interface for the Differential Evolution solver from the scipy package.\n",
    "    Parameters\n",
    "    ----------\n",
    "    objfunc: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    #This is from scipy.optimize.differential_evolution\n",
    "    bounds : sequence\n",
    "        Bounds for variables.  ``(min, max)`` pairs for each element in ``x``,\n",
    "        defining the lower and upper bounds for the optimizing argument of\n",
    "        `func`. It is required to have ``len(bounds) == len(x)``.\n",
    "        ``len(bounds)`` is used to determine the number of parameters in ``x``.\n",
    "    args : tuple, optional\n",
    "        Any additional fixed parameters needed to\n",
    "        completely specify the objective function.\n",
    "    strategy : str, optional\n",
    "        The differential evolution strategy to use. Should be one of:\n",
    "            - 'best1bin'\n",
    "            - 'best1exp'\n",
    "            - 'rand1exp'\n",
    "            - 'randtobest1exp'\n",
    "            - 'best2exp'\n",
    "            - 'rand2exp'\n",
    "            - 'randtobest1bin'\n",
    "            - 'best2bin'\n",
    "            - 'rand2bin'\n",
    "            - 'rand1bin'\n",
    "        The default is 'best1bin'.\n",
    "    maxiter : int, optional\n",
    "        The maximum number of generations over which the entire population is\n",
    "        evolved. The maximum number of function evaluations (with no polishing)\n",
    "        is: ``(maxiter + 1) * popsize * len(x)``\n",
    "    popsize : int, optional\n",
    "        A multiplier for setting the total population size.  The population has\n",
    "        ``popsize * len(x)`` individuals.\n",
    "    tol : float, optional\n",
    "        When the mean of the population energies, multiplied by tol,\n",
    "        divided by the standard deviation of the population energies\n",
    "        is greater than 1 the solving process terminates:\n",
    "        ``convergence = mean(pop) * tol / stdev(pop) > 1``\n",
    "    mutation : float or tuple(float, float), optional\n",
    "        The mutation constant. In the literature this is also known as\n",
    "        differential weight, being denoted by F.\n",
    "        If specified as a float it should be in the range [0, 2].\n",
    "        If specified as a tuple ``(min, max)`` dithering is employed. Dithering\n",
    "        randomly changes the mutation constant on a generation by generation\n",
    "        basis. The mutation constant for that generation is taken from\n",
    "        ``U[min, max)``. Dithering can help speed convergence significantly.\n",
    "        Increasing the mutation constant increases the search radius, but will\n",
    "        slow down convergence.\n",
    "    recombination : float, optional\n",
    "        The recombination constant, should be in the range [0, 1]. In the\n",
    "        literature this is also known as the crossover probability, being\n",
    "        denoted by CR. Increasing this value allows a larger number of mutants\n",
    "        to progress into the next generation, but at the risk of population\n",
    "        stability.\n",
    "    seed : int or `np.random.RandomState`, optional\n",
    "        If `seed` is not specified the `np.RandomState` singleton is used.\n",
    "        If `seed` is an int, a new `np.random.RandomState` instance is used,\n",
    "        seeded with seed.\n",
    "        If `seed` is already a `np.random.RandomState instance`, then that\n",
    "        `np.random.RandomState` instance is used.\n",
    "        Specify `seed` for repeatable minimizations.\n",
    "    disp : bool, optional\n",
    "        Display status messages\n",
    "    callback : callable, `callback(xk, convergence=val)`, optional\n",
    "        A function to follow the progress of the minimization. ``xk`` is\n",
    "        the current value of ``x0``. ``val`` represents the fractional\n",
    "        value of the population convergence.  When ``val`` is greater than one\n",
    "        the function halts. If callback returns `True`, then the minimization\n",
    "        is halted (any polishing is still carried out).\n",
    "    polish : bool, optional\n",
    "        If True (default), then `scipy.optimize.minimize` with the `L-BFGS-B`\n",
    "        method is used to polish the best population member at the end, which\n",
    "        can improve the minimization slightly.\n",
    "    init : string, optional\n",
    "        Specify how the population initialization is performed. Should be\n",
    "        one of:\n",
    "            - 'latinhypercube'\n",
    "            - 'random'\n",
    "        The default is 'latinhypercube'. Latin Hypercube sampling tries to\n",
    "        maximize coverage of the available parameter space. 'random' initializes\n",
    "        the population randomly - this has the drawback that clustering can\n",
    "        occur, preventing the whole of parameter space being covered.\n",
    "    \n",
    "    Attributes\n",
    "    ----------\n",
    "    epoch: int\n",
    "        Current iteration of the selected solver.\n",
    "    pot_func: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    best: (aray,float),  (n_components,),1\n",
    "        Tuple containing the minimum value found for the objective function\n",
    "        and the parameters that yielded that value.\n",
    "    best_pos_so_far: (array), (n_components)\n",
    "        Alias for the best parameters found by the optimizer.\n",
    "    best_so_far: float\n",
    "        Alias for minimum value found by the optimizer.\n",
    "    \"\"\"\n",
    "    def __init__(self,\n",
    "                 objfunc,\n",
    "                 strategy='best1bin',\n",
    "                 maxiter=1000,\n",
    "                 popsize=15,\n",
    "                 max_reads=150000,\n",
    "                 tol=0.01,\n",
    "                 mutation=(0.5, 1),\n",
    "                 recombination=0.7,\n",
    "                 seed=160290,\n",
    "                 callback=None,\n",
    "                 disp=False,\n",
    "                 polish=True,\n",
    "                 init='latinhypercube',\n",
    "                 **kwargs):\n",
    "        \n",
    "        def false():\n",
    "            return False\n",
    "        self.pot_func = objfunc\n",
    "        self.best_pos_so_far = None\n",
    "        self.best_so_far = 1e20\n",
    "        self._ext_callback =lambda x,conv: False if callback is None else callback\n",
    "        self._run_callback = false\n",
    "        self.epoch=0\n",
    "        strats = ['best1bin','best1exp','rand1exp','randtobest1exp','best2exp','rand2exp','randtobest1bin','best2bin','rand2bin','rand1bin']\n",
    "        dash = ['r$N=differential_evolution',[\"##Differential evolution$n=title\",\n",
    "                                              ['r$N=controls_row',[[\"c$N=first_row\",[\"(0,100000,1,\"+str(maxiter)+\")$d=Maxiter&n=max_epoch\",\n",
    "                                                               \"(0,1e8,1,\"+str(max_reads)+\")$d=Max reads\",\n",
    "                                                               \"(1,1e5,1,\"+str(int(popsize))+\")$d=Popsize\"]],\n",
    "                                             \n",
    "                                              [\"c$N=sec_row\",[\"(0.,10.,.01,\"+str(tol)+\")$d=Tol\",\n",
    "                                                               \"(0.,100.,0.1,\"+str(mutation)+\")$d=Mutation\",\n",
    "                                                               \"(0,1e8,1,\"+str(recombination)+\")$d=Recombination\"]\n",
    "                                              ],\n",
    "                                            [\"c$N=third_row\",[\"(0,1000000,1,\"+str(seed)+\")$d=seed\",\n",
    "                                                               \"dd$d=Strategy&o=\"+str(strats)+\"&val=\"+str(strategy),\n",
    "                                                               \"togs$d=Init&o=['latinhypercube','random']\",\n",
    "                                                               ]\n",
    "                                              ],[\"c$n=btn_col\",[\"[\"+str(bool(polish))+\"]$d=Polish\",\"btn$d=Run&n=run_btn\"]]]\n",
    "               ]]]\n",
    "        Dashboard.__init__(self,dash,**kwargs)\n",
    "        self.run_btn.observe(self.run)\n",
    "        \n",
    "    @property\n",
    "    def best(self):\n",
    "        return (self.best_pos_so_far.copy(), float(self.best_so_far))\n",
    "        \n",
    "    \n",
    "    def run(self,_=None, end_callback=None):\n",
    "        \"\"\"Solves the objective funtion\"\"\"\n",
    "        self.reset()\n",
    "        if not end_callback is None:\n",
    "            self._run_callback = end_callback\n",
    "        result = differential_evolution(self.pot_func.evaluate,\n",
    "                                        self.pot_func.domain,\n",
    "                                        callback=self.callback,\n",
    "                                        maxiter=int(self.max_epoch.value),\n",
    "                                        popsize=int(self.popsize.value),\n",
    "                                        tol=self.tol.value,\n",
    "                                        mutation=self.mutation.value,\n",
    "                                        recombination=self.recombination.value,\n",
    "                                        strategy=self.strategy.value,\n",
    "                                        init=self.init.value,\n",
    "                                        polish=self.polish.value)\n",
    "        \n",
    "        clear_output(True)\n",
    "        print(result)\n",
    "        \n",
    "        self.update_best(result.x)\n",
    "\n",
    "    def callback(self,x,convergence):\n",
    "        \"\"\"Callback support for custom stopping of the function\"\"\"\n",
    "        self.update_best(x)\n",
    "        self.epoch += 1\n",
    "        end = self._ext_callback(x,convergence)\n",
    "\n",
    "        return end or self._run_callback() \n",
    "    \n",
    "    def reset(self):\n",
    "        \"\"\"Resets the solver attributes\"\"\"\n",
    "        self.best_pos_so_far = None\n",
    "        self.best_so_far = 1e20\n",
    "        self.epoch=0\n",
    "        self.pot_func.n_reads = 0\n",
    "    \n",
    "    def update_best(self,x):\n",
    "        \"\"\"Keeps track of the best value and parameters found by the solver\"\"\"\n",
    "        val = self.pot_func.evaluate(x)\n",
    "        if self.best_so_far>val:\n",
    "            self.best_pos_so_far = x\n",
    "            self.best_so_far = val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "de = DifferentialEvolution(levy13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "     fun: 3.7946734490167239e-21\n",
      " message: 'Optimization terminated successfully.'\n",
      "    nfev: 1653\n",
      "     nit: 54\n",
      " success: True\n",
      "       x: array([ 1.,  1.])\n"
     ]
    }
   ],
   "source": [
    "de.widget"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"solvers_data/de_widget.png\"></img>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 1.,  1.]), 3.794673449016724e-21)"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "de.best"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "# 3. Solver selector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "No we blended the two dashboards into one, so we are able to choose which algorithm will be use so solve a given objective function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "class Optimizer(Dashboard):\n",
    "    \"\"\"Widget interface to scipy global optimization algorithms.\n",
    "    It allows to select the parameters for a global optimizer.\n",
    "    \n",
    "    Supported algorithms are Basin hopping and Diferential evolution. \n",
    "    This class executes the solver on a function object.\n",
    "    \n",
    "    Attributes\n",
    "    ----------\n",
    "    algos : list, ['differential_evolution','basin_hopping']\n",
    "        Contains the names of the available solvers.\n",
    "    epoch: int\n",
    "        Current iteration of the selected solver.\n",
    "    max_epoch: int\n",
    "        Maximum number of iterations of the selected solver.\n",
    "    max_reads: int\n",
    "        Maximum number of functions reads allowed for the selected solver.\n",
    "    pot_func: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    best: (aray,float),  (n_components,),1\n",
    "        tuple containing the minimum value found for the objective function\n",
    "        and the parameters that yielded that value.\n",
    "    \"\"\"\n",
    "    def __init__(self,objfunc,select='Basin hopping',fractal_kwargs={},basinhopping_kwargs={},diffevo_kwargs={},**kwargs):\n",
    "        \n",
    "        differential_evolution = DifferentialEvolution(objfunc,name='differential_evolution',**diffevo_kwargs)\n",
    "        basin_hopping = BasinHopping(objfunc,name='basin_hopping',**basinhopping_kwargs)\n",
    "\n",
    "        \n",
    "        dash = [\"c$N=optimizer_dash\",[differential_evolution,\n",
    "                                      basin_hopping,\n",
    "                                      [\"r$n=btn_row\",\n",
    "                                        [\"togs$N=algo_sel&o=['Differential evolution','Basin hopping']&val=\"+str(select),\n",
    "                                         'btn$d=Run&n=run_btn']\n",
    "                                      ]\n",
    "                                     ]\n",
    "               ]\n",
    "        \n",
    "        Dashboard.__init__(self,dash,**kwargs)\n",
    "        self.algos = ['differential_evolution','basin_hopping']\n",
    "        self._last_eval = select.lower().replace(' ','_')\n",
    "        self.algo_sel.observe(self.update_layout)\n",
    "        self.run_btn.observe(self.run)\n",
    "        self.update_layout()\n",
    "    \n",
    "    def run(self,_=None, name=None, end_callback=None):\n",
    "        \"\"\"Solves the objective funtion with the currently selected parameters\"\"\"\n",
    "        opt = self.algo_sel.value.lower().replace(' ','_') if name is None else name\n",
    "        self._last_eval = opt\n",
    "        getattr(self,opt).run(end_callback=end_callback)\n",
    "    \n",
    "    @property \n",
    "    def epoch(self):\n",
    "        return getattr(self,self._last_eval).epoch\n",
    "    @epoch.setter\n",
    "    def epoch(self,val):\n",
    "        getattr(self,self._last_eval).epoch = val\n",
    "        \n",
    "    @property \n",
    "    def max_epoch(self):\n",
    "        return getattr(self,self._last_eval).max_epoch\n",
    "    @max_epoch.setter\n",
    "    def max_epoch(self,val):\n",
    "        getattr(self,self._last_eval).max_epoch = val\n",
    "        \n",
    "    @property \n",
    "    def max_reads(self):\n",
    "        return getattr(self,self._last_eval).max_reads\n",
    "    @max_reads.setter\n",
    "    def max_reads(self,val):\n",
    "        getattr(self,self._last_eval).max_reads = val\n",
    "    \n",
    "    @property\n",
    "    def pot_func(self):\n",
    "        return getattr(self,self._last_eval).pot_func\n",
    "    \n",
    "    @pot_func.setter\n",
    "    def pot_func(self,val):\n",
    "        getattr(self,self._last_eval).pot_func = val\n",
    "    \n",
    "    @property\n",
    "    def best(self):\n",
    "        optimizer = getattr(self,self._last_eval)\n",
    "        return (optimizer.best_pos_so_far, float(optimizer.best_so_far))\n",
    "    \n",
    "    def reset(self):\n",
    "        \"\"\"Reset the solvers\"\"\"\n",
    "        self.basin_hopping.reset()\n",
    "        self.differential_evolution.reset()\n",
    "    \n",
    "    def update_layout(self,_=None):\n",
    "        \"\"\"widget interface updating\"\"\"\n",
    "        if self.algo_sel.value == \"Differential evolution\":\n",
    "            self.differential_evolution.visible = True\n",
    "            self.differential_evolution.run_btn.visible = False\n",
    "            self.basin_hopping.visible = False\n",
    "          \n",
    "        elif self.algo_sel.value == \"Basin hopping\":\n",
    "            self.differential_evolution.visible = False\n",
    "            self.basin_hopping.visible = True\n",
    "            self.basin_hopping.run_btn.visible = False\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": [
    "solvers = Optimizer(levy13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                        fun: 9.531626804523418e-17\n",
      " lowest_optimization_result:       fun: 9.531626804523418e-17\n",
      " hess_inv: array([[  5.46483894e-03,  -7.99089436e-05],\n",
      "       [ -7.99089436e-05,   4.99906990e-01]])\n",
      "      jac: array([  1.15863571e-06,   1.03142188e-08])\n",
      "  message: 'Optimization terminated successfully.'\n",
      "     nfev: 48\n",
      "      nit: 8\n",
      "     njev: 12\n",
      "   status: 0\n",
      "  success: True\n",
      "        x: array([ 1.,  1.])\n",
      "                    message: ['requested number of basinhopping iterations completed successfully']\n",
      "      minimization_failures: 0\n",
      "                       nfev: 5672\n",
      "                        nit: 100\n",
      "                       njev: 1418\n",
      "                          x: array([ 1.,  1.])\n"
     ]
    }
   ],
   "source": [
    "solvers.widget"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": true,
    "editable": true
   },
   "source": [
    "<img src=\"solvers_data/de_widget.png\"></img>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 1.,  1.]), 9.531626804523418e-17)"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "solvers.best"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false,
    "deletable": true,
    "editable": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting optimizer.py\n"
     ]
    }
   ],
   "source": [
    "%%file optimizer.py\n",
    "import time#\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import random\n",
    "import copy\n",
    "\n",
    "from scipy.optimize import  differential_evolution,minimize\n",
    "from scipy.optimize import basinhopping\n",
    "from scipy.optimize import  differential_evolution\n",
    "\n",
    "from IPython.core.display import clear_output\n",
    "\n",
    "from shaolin.core.dashboard import Dashboard\n",
    "from test_functions import TestFunctions\n",
    "\n",
    "\"\"\"\n",
    "Created on Tue May 24 13:13:28 2016\n",
    "\n",
    "@author: Hossam Faris\n",
    "\"\"\"\n",
    "import math\n",
    "import numpy\n",
    "import random\n",
    "import time\n",
    "\n",
    "\n",
    "class solution:\n",
    "\n",
    "    def __init__(self):\n",
    "        self.endTime=None\n",
    "        self.executionTime=None\n",
    "        self.convergence=None\n",
    "        self.optimizer=None\n",
    "        self.objfname=None\n",
    "        self.fmin = None\n",
    "        self.nest = None\n",
    "            \n",
    "def get_cuckoos(nest,best,domain,n,dim):\n",
    "    \n",
    "    # perform Levy flights\n",
    "    tempnest=numpy.zeros((n,dim))\n",
    "    tempnest=numpy.array(nest)\n",
    "    beta=3/2;\n",
    "    sigma=(math.gamma(1+beta)*math.sin(math.pi*beta/2)/(math.gamma((1+beta)/2)*beta*2**((beta-1)/2)))**(1/beta);\n",
    "\n",
    "    s=numpy.zeros(dim)\n",
    "    for j in range (0,n):\n",
    "        s=nest[j,:]\n",
    "        u=numpy.random.randn(len(s))*sigma\n",
    "        v=numpy.random.randn(len(s))\n",
    "        step=u/abs(v)**(1/beta)\n",
    " \n",
    "        stepsize=0.01*(step*(s-best))\n",
    "\n",
    "        s=s+stepsize*numpy.random.randn(len(s))\n",
    "    \n",
    "        for i,(lb,ub) in enumerate(domain):\n",
    "            tempnest[j,i]=numpy.clip(s[i], lb, ub)\n",
    "    return tempnest\n",
    "\n",
    "def get_best_nest(nest,newnest,fitness,n,dim,objf):\n",
    "# Evaluating all new solutions\n",
    "    tempnest=numpy.zeros((n,dim))\n",
    "    tempnest=numpy.copy(nest)\n",
    "\n",
    "    for j in range(0,n):\n",
    "    #for j=1:size(nest,1),\n",
    "   \n",
    "        fnew=objf.evaluate(newnest[j,:])\n",
    "        #print(fnew)\n",
    "        if fnew<=fitness[j] and objf.in_domain(newnest[j,:]):\n",
    "           fitness[j]=fnew\n",
    "           tempnest[j,:]=newnest[j,:]\n",
    "        \n",
    "    # Find the current best\n",
    "\n",
    "    fmin = min(fitness)\n",
    "    K=numpy.argmin(fitness)\n",
    "    bestlocal=tempnest[K,:]\n",
    "\n",
    "    return fmin,bestlocal,tempnest,fitness\n",
    "\n",
    "# Replace some nests by constructing new solutions/nests\n",
    "def empty_nests(nest,pa,n,dim):\n",
    "\n",
    "    # Discovered or not \n",
    "    tempnest=numpy.zeros((n,dim))\n",
    "\n",
    "    K=numpy.random.uniform(0,1,(n,dim))>pa\n",
    "    \n",
    "    \n",
    "    stepsize=random.random()*(nest[numpy.random.permutation(n),:]-nest[numpy.random.permutation(n),:])\n",
    "\n",
    "    \n",
    "    tempnest=nest+stepsize*K\n",
    " \n",
    "    return tempnest\n",
    "##########################################################################\n",
    "\n",
    "\n",
    "def CS(objf,lb,ub,dim,n,N_IterTotal,callback):\n",
    "\n",
    "    pa=0.25\n",
    "    nd=dim\n",
    "    convergence=[]\n",
    "    # RInitialize nests randomely\n",
    "    nest=numpy.random.rand(n,dim)*(ub-lb)+lb\n",
    "    new_nest=numpy.zeros((n,dim))\n",
    "    new_nest=numpy.copy(nest)\n",
    "    bestnest=[0]*dim;\n",
    "    fitness=numpy.zeros(n) \n",
    "    fitness.fill(float(\"inf\"))\n",
    "    s=solution()\n",
    "    timerStart=time.time() \n",
    "    s.startTime=time.strftime(\"%Y-%m-%d-%H-%M-%S\")\n",
    "\n",
    "    fmin,bestnest,nest,fitness =get_best_nest(nest,new_nest,fitness,n,dim,objf)\n",
    "    convergence = [];\n",
    "    # Main loop counter\n",
    "    iter = 0\n",
    "    while iter <= N_IterTotal and not callback() and (fmin-objf.benchmark[1])>10**-6:\n",
    "    # Generate new solutions (but keep the current best\n",
    "        new_nest=get_cuckoos(nest,bestnest,objf.domain,n,dim)\n",
    "        # Evaluate new solutions and find best\n",
    "        fnew,best,nest,fitness=get_best_nest(nest,new_nest,fitness,n,dim,objf)\n",
    "        new_nest=empty_nests(new_nest,pa,n,dim) ;\n",
    "        # Evaluate new solutions and find best\n",
    "        fnew,best,nest,fitness=get_best_nest(nest,new_nest,fitness,n,dim,objf)\n",
    "        if (iter%100==0):\n",
    "            result = minimize(objf.evaluate,\n",
    "                              bestnest,\n",
    "                              bounds=objf.domain,\n",
    "                              method=\"L-BFGS-B\",\n",
    "                             )\n",
    "            best = result.x\n",
    "            fnew = result.fun\n",
    "        if fnew<fmin and objf.in_domain(best):\n",
    "            fmin=fnew\n",
    "            bestnest=best\n",
    "            s.fmin = fmin\n",
    "            s.nest = bestnest\n",
    "        convergence.append(fmin)\n",
    "        iter += 1\n",
    "\n",
    "    timerEnd=time.time()  \n",
    "    s.endTime=time.strftime(\"%Y-%m-%d-%H-%M-%S\")\n",
    "    s.executionTime=timerEnd-timerStart\n",
    "    s.convergence=convergence\n",
    "    s.optimizer=\"CS\"\n",
    "    s.objfname=objf.evaluate.__name__\n",
    "    s.fmin = fmin\n",
    "    s.nest = bestnest\n",
    "    print(\"best value found: {}\".format(fmin))\n",
    "    return s\n",
    "\n",
    "from shaolin import KungFu\n",
    "import numpy as np\n",
    "class CuckooSearch(KungFu):\n",
    "    \n",
    "    def __init__(self,objfunc,num_iters=500,n_birds=50,callback=None,max_reads=50000,**kwargs):\n",
    "        \n",
    "        self.num_iters = num_iters\n",
    "        self.n_birds = n_birds\n",
    "        self.pot_func = objfunc\n",
    "        def false():\n",
    "            return False\n",
    "        self.pot_func = objfunc\n",
    "        #self.best_pos_so_far = None\n",
    "        #self.best_so_far = 1e20\n",
    "        self._ext_callback =lambda x,conv: False if callback is None else callback\n",
    "        self._run_callback = false\n",
    "        self.epoch=0\n",
    "        self.solution = solution()\n",
    "        self.solution.nest = np.zeros(len(self.pot_func.random_in_domain()))\n",
    "        self.solution.fmin = 1e20\n",
    "        KungFu.__init__(self,**{'title':'#Cuckoo search$N=title&d=',\n",
    "                         'num_iters':(0,1000000,1,num_iters),\n",
    "                         'max_reads':(0,1000000,1,max_reads),\n",
    "                         'n_birds':(0,1000000,1,n_birds),\n",
    "                         'run_button':'@btn$N=run_btn&d=Run'\n",
    "                        },box='1r|',**kwargs)\n",
    "    \n",
    "    @property\n",
    "    def best(self):\n",
    "        return self.solution.nest,self.solution.fmin\n",
    "    @property\n",
    "    def best_pos_so_far(self):\n",
    "        return self.solution.nest\n",
    "    @property\n",
    "    def best_so_far(self):\n",
    "        return self.solution.fmin\n",
    "    \n",
    "    def reset(self):\n",
    "        self.solution.nest = np.zeros(len(self.pot_func.random_in_domain()))\n",
    "        self.solution.fmin = 1e20\n",
    "        self.epoch=0\n",
    "        self.pot_func.n_reads = 0\n",
    "    \n",
    "    def run(self,_=None, end_callback=None):\n",
    "        #print('runnin')\n",
    "        #self.solution.nest = np.zeros(len(self.pot_func.random_in_domain()))\n",
    "        dim = len(self.pot_func.random_in_domain())\n",
    "        ub = np.asarray(self.pot_func.domain).max()\n",
    "        lb = np.asarray(self.pot_func.domain).min()\n",
    "        \n",
    "        self.solution = CS(self.pot_func,lb,ub,dim,self.n_birds(),self.num_iters(),end_callback)\n",
    "        \n",
    "    def callback(self,x,convergence):\n",
    "        \"\"\"Callback support for custom stopping of the function\"\"\"\n",
    "        self.epoch += 1\n",
    "        end = self._ext_callback(x,convergence)\n",
    "\n",
    "        return end or self._run_callback() \n",
    "\n",
    "class Optimizer(Dashboard):\n",
    "    \"\"\"Widget interface to scipy global optimization algorithms.\n",
    "    It allows to select the parameters for a global optimizer.\n",
    "    \n",
    "    Supported algorithms are Basin hopping and Diferential evolution. \n",
    "    This class executes the solver on a function object.\n",
    "    \n",
    "    Attributes\n",
    "    ----------\n",
    "    algos : list, ['differential_evolution','basin_hopping']\n",
    "        Contains the names of the available solvers.\n",
    "    epoch: int\n",
    "        Current iteration of the selected solver.\n",
    "    max_epoch: int\n",
    "        Maximum number of iterations of the selected solver.\n",
    "    max_reads: int\n",
    "        Maximum number of functions reads allowed for the selected solver.\n",
    "    pot_func: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    best: (aray,float),  (n_components,),1\n",
    "        tuple containing the minimum value found for the objective function\n",
    "        and the parameters that yielded that value.\n",
    "    \"\"\"\n",
    "    def __init__(self,objfunc,select='Cuckoo search',fractal_kwargs={},basinhopping_kwargs={},diffevo_kwargs={},**kwargs):\n",
    "        \n",
    "        differential_evolution = DifferentialEvolution(objfunc,name='differential_evolution',**diffevo_kwargs)\n",
    "        basin_hopping = BasinHopping(objfunc,name='basin_hopping',**basinhopping_kwargs)\n",
    "        cuckoo = CuckooSearch(objfunc=objfunc,name='cuckoo_search')\n",
    "        \n",
    "        dash = [\"c$N=optimizer_dash\",[differential_evolution,\n",
    "                                      basin_hopping,\n",
    "                                      cuckoo,\n",
    "                                      [\"r$n=btn_row\",\n",
    "                                        [\"togs$N=algo_sel&o=['Cuckoo search','Differential evolution','Basin hopping']&val=\"+str(select),\n",
    "                                         'btn$d=Run&n=run_btn']\n",
    "                                      ]\n",
    "                                     ]\n",
    "               ]\n",
    "        \n",
    "        Dashboard.__init__(self,dash,**kwargs)\n",
    "        self.algos = ['cuckoo_search','differential_evolution','basin_hopping']\n",
    "        self._last_eval = select.lower().replace(' ','_')\n",
    "        self.algo_sel.observe(self.update_layout)\n",
    "        self.run_btn.observe(self.run)\n",
    "        #self.cuckoo_search.run_btn.observe(self.run)\n",
    "        self.update_layout()\n",
    "    \n",
    "    def run(self,_=None, name=None, end_callback=None):\n",
    "        \"\"\"Solves the objective funtion with the currently selected parameters\"\"\"\n",
    "        opt = self.algo_sel.value.lower().replace(' ','_') if name is None else name\n",
    "        self._last_eval = opt\n",
    "        getattr(self,opt).run(end_callback=end_callback)\n",
    "    \n",
    "    @property \n",
    "    def epoch(self):\n",
    "        return getattr(self,self._last_eval).epoch\n",
    "    @epoch.setter\n",
    "    def epoch(self,val):\n",
    "        getattr(self,self._last_eval).epoch = val\n",
    "        \n",
    "    @property \n",
    "    def max_epoch(self):\n",
    "        return getattr(self,self._last_eval).max_epoch\n",
    "    @max_epoch.setter\n",
    "    def max_epoch(self,val):\n",
    "        getattr(self,self._last_eval).max_epoch = val\n",
    "        \n",
    "    @property \n",
    "    def max_reads(self):\n",
    "        return getattr(self,self._last_eval).max_reads\n",
    "    @max_reads.setter\n",
    "    def max_reads(self,val):\n",
    "        getattr(self,self._last_eval).max_reads = val\n",
    "    \n",
    "    @property\n",
    "    def pot_func(self):\n",
    "        return getattr(self,self._last_eval).pot_func\n",
    "    \n",
    "    @pot_func.setter\n",
    "    def pot_func(self,val):\n",
    "        getattr(self,self._last_eval).pot_func = val\n",
    "    \n",
    "    @property\n",
    "    def best(self):\n",
    "        optimizer = getattr(self,self._last_eval)\n",
    "        return (optimizer.best_pos_so_far, float(optimizer.best_so_far))\n",
    "    \n",
    "    def reset(self):\n",
    "        \"\"\"Reset the solvers\"\"\"\n",
    "        self.basin_hopping.reset()\n",
    "        self.differential_evolution.reset()\n",
    "        self.cuckoo_search.reset()\n",
    "    \n",
    "    def update_layout(self,_=None):\n",
    "        \"\"\"widget interface updating\"\"\"\n",
    "        if self.algo_sel.value == \"Differential evolution\":\n",
    "            self.differential_evolution.visible = True\n",
    "            self.differential_evolution.run_btn.visible = True\n",
    "            self.basin_hopping.visible = False\n",
    "            self.basin_hopping.run_btn.visible = False\n",
    "            self.cuckoo_search.run_btn.visible = False\n",
    "            self.cuckoo_search.visible = False\n",
    "          \n",
    "        elif self.algo_sel.value == \"Basin hopping\":\n",
    "            self.differential_evolution.visible = False\n",
    "            self.basin_hopping.visible = True\n",
    "            self.basin_hopping.run_btn.visible = True\n",
    "            self.cuckoo_search.run_btn.visible = False\n",
    "            self.cuckoo_search.visible = False\n",
    "            \n",
    "        elif self.algo_sel.value == 'Cuckoo search':\n",
    "            self.differential_evolution.visible = False\n",
    "            self.differential_evolution.run_btn.visible = False\n",
    "            self.basin_hopping.visible = False\n",
    "            self.basin_hopping.run_btn.visible = False\n",
    "            self.cuckoo_search.run_btn.visible = False\n",
    "            self.cuckoo_search.visible = True\n",
    "\n",
    "\n",
    "class DifferentialEvolution(Dashboard):\n",
    "    \"\"\"Shaolin interface for the Differential Evolution solver from the scipy package.\n",
    "    Parameters\n",
    "    ----------\n",
    "    objfunc: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    #This is from scipy.optimize.differential_evolution\n",
    "    bounds : sequence\n",
    "        Bounds for variables.  ``(min, max)`` pairs for each element in ``x``,\n",
    "        defining the lower and upper bounds for the optimizing argument of\n",
    "        `func`. It is required to have ``len(bounds) == len(x)``.\n",
    "        ``len(bounds)`` is used to determine the number of parameters in ``x``.\n",
    "    args : tuple, optional\n",
    "        Any additional fixed parameters needed to\n",
    "        completely specify the objective function.\n",
    "    strategy : str, optional\n",
    "        The differential evolution strategy to use. Should be one of:\n",
    "            - 'best1bin'\n",
    "            - 'best1exp'\n",
    "            - 'rand1exp'\n",
    "            - 'randtobest1exp'\n",
    "            - 'best2exp'\n",
    "            - 'rand2exp'\n",
    "            - 'randtobest1bin'\n",
    "            - 'best2bin'\n",
    "            - 'rand2bin'\n",
    "            - 'rand1bin'\n",
    "        The default is 'best1bin'.\n",
    "    maxiter : int, optional\n",
    "        The maximum number of generations over which the entire population is\n",
    "        evolved. The maximum number of function evaluations (with no polishing)\n",
    "        is: ``(maxiter + 1) * popsize * len(x)``\n",
    "    popsize : int, optional\n",
    "        A multiplier for setting the total population size.  The population has\n",
    "        ``popsize * len(x)`` individuals.\n",
    "    tol : float, optional\n",
    "        When the mean of the population energies, multiplied by tol,\n",
    "        divided by the standard deviation of the population energies\n",
    "        is greater than 1 the solving process terminates:\n",
    "        ``convergence = mean(pop) * tol / stdev(pop) > 1``\n",
    "    mutation : float or tuple(float, float), optional\n",
    "        The mutation constant. In the literature this is also known as\n",
    "        differential weight, being denoted by F.\n",
    "        If specified as a float it should be in the range [0, 2].\n",
    "        If specified as a tuple ``(min, max)`` dithering is employed. Dithering\n",
    "        randomly changes the mutation constant on a generation by generation\n",
    "        basis. The mutation constant for that generation is taken from\n",
    "        ``U[min, max)``. Dithering can help speed convergence significantly.\n",
    "        Increasing the mutation constant increases the search radius, but will\n",
    "        slow down convergence.\n",
    "    recombination : float, optional\n",
    "        The recombination constant, should be in the range [0, 1]. In the\n",
    "        literature this is also known as the crossover probability, being\n",
    "        denoted by CR. Increasing this value allows a larger number of mutants\n",
    "        to progress into the next generation, but at the risk of population\n",
    "        stability.\n",
    "    seed : int or `np.random.RandomState`, optional\n",
    "        If `seed` is not specified the `np.RandomState` singleton is used.\n",
    "        If `seed` is an int, a new `np.random.RandomState` instance is used,\n",
    "        seeded with seed.\n",
    "        If `seed` is already a `np.random.RandomState instance`, then that\n",
    "        `np.random.RandomState` instance is used.\n",
    "        Specify `seed` for repeatable minimizations.\n",
    "    disp : bool, optional\n",
    "        Display status messages\n",
    "    callback : callable, `callback(xk, convergence=val)`, optional\n",
    "        A function to follow the progress of the minimization. ``xk`` is\n",
    "        the current value of ``x0``. ``val`` represents the fractional\n",
    "        value of the population convergence.  When ``val`` is greater than one\n",
    "        the function halts. If callback returns `True`, then the minimization\n",
    "        is halted (any polishing is still carried out).\n",
    "    polish : bool, optional\n",
    "        If True (default), then `scipy.optimize.minimize` with the `L-BFGS-B`\n",
    "        method is used to polish the best population member at the end, which\n",
    "        can improve the minimization slightly.\n",
    "    init : string, optional\n",
    "        Specify how the population initialization is performed. Should be\n",
    "        one of:\n",
    "            - 'latinhypercube'\n",
    "            - 'random'\n",
    "        The default is 'latinhypercube'. Latin Hypercube sampling tries to\n",
    "        maximize coverage of the available parameter space. 'random' initializes\n",
    "        the population randomly - this has the drawback that clustering can\n",
    "        occur, preventing the whole of parameter space being covered.\n",
    "    \n",
    "    Attributes\n",
    "    ----------\n",
    "    epoch: int\n",
    "        Current iteration of the selected solver.\n",
    "    pot_func: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    best: (aray,float),  (n_components,),1\n",
    "        Tuple containing the minimum value found for the objective function\n",
    "        and the parameters that yielded that value.\n",
    "    best_pos_so_far: (array), (n_components)\n",
    "        Alias for the best parameters found by the optimizer.\n",
    "    best_so_far: float\n",
    "        Alias for minimum value found by the optimizer.\n",
    "    \"\"\"\n",
    "    def __init__(self,\n",
    "                 objfunc,\n",
    "                 strategy='best1bin',\n",
    "                 maxiter=1000,\n",
    "                 popsize=15,\n",
    "                 max_reads=150000,\n",
    "                 tol=0.01,\n",
    "                 mutation=(0.5, 1),\n",
    "                 recombination=0.7,\n",
    "                 seed=160290,\n",
    "                 callback=None,\n",
    "                 disp=False,\n",
    "                 polish=True,\n",
    "                 init='latinhypercube',\n",
    "                 **kwargs):\n",
    "        \n",
    "        def false():\n",
    "            return False\n",
    "        self.pot_func = objfunc\n",
    "        self.best_pos_so_far = None\n",
    "        self.best_so_far = 1e20\n",
    "        self._ext_callback =lambda x,conv: False if callback is None else callback\n",
    "        self._run_callback = false\n",
    "        self.epoch=0\n",
    "        strats = ['best1bin','best1exp','rand1exp','randtobest1exp','best2exp','rand2exp','randtobest1bin','best2bin','rand2bin','rand1bin']\n",
    "        dash = ['r$N=differential_evolution',[\"##Differential evolution$n=title\",\n",
    "                                              ['r$N=controls_row',[[\"c$N=first_row\",[\"(0,100000,1,\"+str(maxiter)+\")$d=Maxiter&n=max_epoch\",\n",
    "                                                               \"(0,1e8,1,\"+str(max_reads)+\")$d=Max reads\",\n",
    "                                                               \"(1,1e5,1,\"+str(int(popsize))+\")$d=Popsize\"]],\n",
    "                                             \n",
    "                                              [\"c$N=sec_row\",[\"(0.,10.,.01,\"+str(tol)+\")$d=Tol\",\n",
    "                                                               \"(0.,100.,0.1,\"+str(mutation)+\")$d=Mutation\",\n",
    "                                                               \"(0,1e8,1,\"+str(recombination)+\")$d=Recombination\"]\n",
    "                                              ],\n",
    "                                            [\"c$N=third_row\",[\"(0,1000000,1,\"+str(seed)+\")$d=seed\",\n",
    "                                                               \"dd$d=Strategy&o=\"+str(strats)+\"&val=\"+str(strategy),\n",
    "                                                               \"togs$d=Init&o=['latinhypercube','random']\",\n",
    "                                                               ]\n",
    "                                              ],[\"c$n=btn_col\",[\"[\"+str(bool(polish))+\"]$d=Polish\",\"btn$d=Run&n=run_btn\"]]]\n",
    "               ]]]\n",
    "        Dashboard.__init__(self,dash,**kwargs)\n",
    "        self.run_btn.observe(self.run)\n",
    "        \n",
    "    @property\n",
    "    def best(self):\n",
    "        return (self.best_pos_so_far.copy(), float(self.best_so_far))\n",
    "        \n",
    "    \n",
    "    def run(self,_=None, end_callback=None):\n",
    "        \"\"\"Solves the objective funtion\"\"\"\n",
    "        self.reset()\n",
    "        if not end_callback is None:\n",
    "            self._run_callback = end_callback\n",
    "        result = differential_evolution(self.pot_func.evaluate,\n",
    "                                        self.pot_func.domain,\n",
    "                                        callback=self.callback,\n",
    "                                        maxiter=int(self.max_epoch.value),\n",
    "                                        popsize=int(self.popsize.value),\n",
    "                                        tol=self.tol.value,\n",
    "                                        mutation=self.mutation.value,\n",
    "                                        recombination=self.recombination.value,\n",
    "                                        strategy=self.strategy.value,\n",
    "                                        init=self.init.value,\n",
    "                                        polish=self.polish.value)\n",
    "        \n",
    "        clear_output(True)\n",
    "        print(result)\n",
    "        \n",
    "        self.update_best(result.x)\n",
    "\n",
    "    def callback(self,x,convergence):\n",
    "        \"\"\"Callback support for custom stopping of the function\"\"\"\n",
    "        self.update_best(x)\n",
    "        self.epoch += 1\n",
    "        end = self._ext_callback(x,convergence)\n",
    "\n",
    "        return end or self._run_callback() \n",
    "    \n",
    "    def reset(self):\n",
    "        \"\"\"Resets the solver attributes\"\"\"\n",
    "        self.best_pos_so_far = None\n",
    "        self.best_so_far = 1e20\n",
    "        self.epoch=0\n",
    "        self.pot_func.n_reads = 0\n",
    "    \n",
    "    def update_best(self,x):\n",
    "        \"\"\"Keeps track of the best value and parameters found by the solver\"\"\"\n",
    "        val = self.pot_func.evaluate(x)\n",
    "        if self.best_so_far>val:\n",
    "            self.best_pos_so_far = x\n",
    "            self.best_so_far = val\n",
    "\n",
    "class MyBounds(object):\n",
    "    \"\"\"Class in charge of managing the boundaries of the sampling region for the basin hopping solver.\"\"\"\n",
    "    def __init__(self, xmax=[1.1,1.1], xmin=[-1.1,-1.1],other_test=None ):\n",
    "        def true(**kwargs):\n",
    "            return True\n",
    "        self._outer_accept_test = true if other_test is None else other_test\n",
    "        self.xmax = np.array(xmax)\n",
    "        self.xmin = np.array(xmin)\n",
    "\n",
    "    def __call__(self, **kwargs):\n",
    "        x = kwargs[\"x_new\"]\n",
    "        out = self._outer_accept_test(**kwargs)\n",
    "        tmax = bool(np.all(x <= self.xmax))\n",
    "        tmin = bool(np.all(x >= self.xmin))\n",
    "        return tmax and tmin and out\n",
    "\n",
    "class BasinHopping(Dashboard):\n",
    "    \"\"\"Find the global minimum of a function using the basin-hopping algorithm\n",
    "    \n",
    "    Parameters\n",
    "    ----------\n",
    "    objfunc: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    x0 : ndarray\n",
    "        Initial guess.\n",
    "    niter : integer, optional\n",
    "        The number of basin hopping iterations\n",
    "    T : float, optional\n",
    "        The \"temperature\" parameter for the accept or reject criterion.  Higher\n",
    "        \"temperatures\" mean that larger jumps in function value will be\n",
    "        accepted.  For best results ``T`` should be comparable to the\n",
    "        separation\n",
    "        (in function value) between local minima.\n",
    "    stepsize : float, optional\n",
    "        initial step size for use in the random displacement.\n",
    "    minimizer_kwargs : dict, optional\n",
    "        Extra keyword arguments to be passed to the minimizer\n",
    "        ``scipy.optimize.minimize()`` Some important options could be:\n",
    "            method : str\n",
    "                The minimization method (e.g. ``\"L-BFGS-B\"``)\n",
    "            args : tuple\n",
    "                Extra arguments passed to the objective function (``func``) and\n",
    "                its derivatives (Jacobian, Hessian).\n",
    "    take_step : callable ``take_step(x)``, optional\n",
    "        Replace the default step taking routine with this routine.  The default\n",
    "        step taking routine is a random displacement of the coordinates, but\n",
    "        other step taking algorithms may be better for some systems.\n",
    "        ``take_step`` can optionally have the attribute ``take_step.stepsize``.\n",
    "        If this attribute exists, then ``basinhopping`` will adjust\n",
    "        ``take_step.stepsize`` in order to try to optimize the global minimum\n",
    "        search.\n",
    "    accept_test : callable, ``accept_test(f_new=f_new, x_new=x_new, f_old=fold, x_old=x_old)``, optional\n",
    "        Define a test which will be used to judge whether or not to accept the\n",
    "        step.  This will be used in addition to the Metropolis test based on\n",
    "        \"temperature\" ``T``.  The acceptable return values are True,\n",
    "        False, or ``\"force accept\"``. If any of the tests return False\n",
    "        then the step is rejected. If the latter, then this will override any\n",
    "        other tests in order to accept the step. This can be used, for example,\n",
    "        to forcefully escape from a local minimum that ``basinhopping`` is\n",
    "        trapped in.\n",
    "    callback : callable, ``callback(x, f, accept)``, optional\n",
    "        A callback function which will be called for all minima found.  ``x``\n",
    "        and ``f`` are the coordinates and function value of the trial minimum,\n",
    "        and ``accept`` is whether or not that minimum was accepted.  This can be\n",
    "        used, for example, to save the lowest N minima found.  Also,\n",
    "        ``callback`` can be used to specify a user defined stop criterion by\n",
    "        optionally returning True to stop the ``basinhopping`` routine.\n",
    "    interval : integer, optional\n",
    "        interval for how often to update the ``stepsize``\n",
    "    disp : bool, optional\n",
    "        Set to True to print status messages\n",
    "    niter_success : integer, optional\n",
    "        Stop the run if the global minimum candidate remains the same for this\n",
    "        number of iterations.\n",
    "        \n",
    "        \n",
    "    Attributes\n",
    "    ----------\n",
    "    epoch: int\n",
    "        Current iteration of the selected solver.\n",
    "    pot_func: ObjectiveFunction object.\n",
    "        Object representing the objective function to be solved. \n",
    "    best: (aray,float),  (n_components,),1\n",
    "        Tuple containing the minimum value found for the objective function\n",
    "        and the parameters that yielded that value.\n",
    "    best_pos_so_far: (array), (n_components)\n",
    "        Alias for the best parameters found by the optimizer.\n",
    "    best_so_far: float\n",
    "        Alias for minimum value found by the optimizer.\n",
    "        \"\"\"\n",
    "    def __init__(self,\n",
    "                 objfunc,\n",
    "                 x0=None,\n",
    "                 max_reads=10000000,\n",
    "                 niter=100,\n",
    "                 T=1.0,\n",
    "                 stepsize=0.5,\n",
    "                 minimizer_kwargs=None,\n",
    "                 take_step=None,\n",
    "                 accept_test=None,\n",
    "                 callback=None,\n",
    "                 interval=50,\n",
    "                 disp=False,\n",
    "                 niter_success=None,\n",
    "                 **kwargs):\n",
    "        def false(**kwargs):\n",
    "            return False\n",
    "        def true(**kwargs):\n",
    "            return True\n",
    "        self._outer_accept_test = true if accept_test is None else accept_test\n",
    "        self.minimizer_kwargs = minimizer_kwargs\n",
    "        self.take_step = take_step\n",
    "        \n",
    "       \n",
    "        self.pot_func = objfunc\n",
    "        self.x0 = x0\n",
    "        self.best_pos_so_far = None\n",
    "        self.best_so_far = 1e20\n",
    "        self._ext_callback =lambda x,val,accept: False if callback is None else callback\n",
    "        self._run_callback = false\n",
    "        self.epoch=0\n",
    "        niter_success = 0 if niter_success is None else niter_success\n",
    "        strats = ['best1bin','best1exp','rand1exp','randtobest1exp','best2exp','rand2exp','randtobest1bin','best2bin','rand2bin','rand1bin']\n",
    "        dash = ['r$N=differential_evolution',[\"##Basin-hopping$n=title\",\n",
    "                                              ['r$N=controls_row',[[\"c$N=first_row\",[\"(0,100000,1,\"+str(niter)+\")$d=Niter&n=max_epoch\",\n",
    "                                                               \"(0,1e8,1,\"+str(max_reads)+\")$d=Max reads\",\n",
    "                                                               \"(0,1e4,1,\"+str(int(niter_success))+\")$d=niter_success\"]],\n",
    "                                             \n",
    "                                              [\"c$N=sec_row\",[\"(0.,10000.,.01,\"+str(T)+\")$d=T\",\n",
    "                                                              \"(0,1e8,1,\"+str(interval)+\")$d=Interval\",\n",
    "                                                               \"(0.,100.,0.01,\"+str(stepsize)+\")$d=Stepsize\"]\n",
    "                                              ],\n",
    "                                              [\"c$n=btn_col\",[\"[\"+str(bool(disp))+\"]$d=Disp\",\"btn$d=Run&n=run_btn\"]]]\n",
    "               ]]]\n",
    "        Dashboard.__init__(self,dash,**kwargs)\n",
    "        self.run_btn.observe(self.run)\n",
    "    \n",
    "    \n",
    "    def accept_test(self,**kwargs):\n",
    "        \"\"\"Custom acceptance test support\"\"\"\n",
    "        out = self._outer_accept_test(**kwargs)\n",
    "        return out or self.pot_func.in_domain(kwargs['x'])\n",
    "    \n",
    "    \n",
    "    \n",
    "    @property\n",
    "    def best(self):\n",
    "        return (self.best_pos_so_far.copy(), float(self.best_so_far))\n",
    "    \n",
    "        \n",
    "    \n",
    "    def run(self,_=None, end_callback=None):\n",
    "        \"\"\"Solves the objective function\"\"\"\n",
    "        self.reset()\n",
    "        if not end_callback is None:\n",
    "            self._run_callback = end_callback\n",
    "        niter_success = None if self.niter_success.value == 0 else int(self.niter_success.value)\n",
    "        x0 = self.pot_func.random_in_domain()\n",
    "        \n",
    "        xmax = [x[1] for x in self.pot_func.domain]\n",
    "        xmin = [x[0] for x in self.pot_func.domain]\n",
    "        \n",
    "        bounds = MyBounds(xmax=xmax,xmin=xmin,other_test=self._outer_accept_test)\n",
    "        result = basinhopping(self.pot_func.evaluate,\n",
    "                              x0,\n",
    "                              niter=self.max_epoch.value,\n",
    "                              T=self.t.value,\n",
    "                              stepsize=self.stepsize.value,\n",
    "                              minimizer_kwargs=self.minimizer_kwargs,\n",
    "                              take_step=self.take_step,\n",
    "                              accept_test=bounds,\n",
    "                              callback=self.callback,\n",
    "                              interval=self.interval.value,\n",
    "                              disp=self.disp.value,\n",
    "                              niter_success=niter_success)\n",
    "        \n",
    "        clear_output(True)\n",
    "        print(result)\n",
    "        self.update_best(result.x)\n",
    "\n",
    "    def callback(self,x,val,accept):\n",
    "        \"\"\"custom callback for finishing the optimization in a scipy-like manner.\"\"\"\n",
    "        self.update_best(x)\n",
    "        self.epoch += 1\n",
    "        end = self._ext_callback(x,val,accept)\n",
    "        return end or self._run_callback() \n",
    "    \n",
    "    def reset(self):\n",
    "        \"\"\"Resets the solver attributes\"\"\"\n",
    "        self.best_pos_so_far = None\n",
    "        self.best_so_far = 1e20\n",
    "        self.epoch=0\n",
    "        self.pot_func.n_reads = 0\n",
    "    \n",
    "    def update_best(self,x):\n",
    "        \"\"\"Keeps track of the best value and parameters found by the solver\"\"\"\n",
    "        val = self.pot_func.evaluate(x)\n",
    "        if self.best_so_far>val:\n",
    "            self.best_pos_so_far = x\n",
    "            self.best_so_far = val"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true,
    "deletable": true,
    "editable": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.3"
  },
  "widgets": {
   "state": {
    "225fa569068647b5a04fd95f63a868c9": {
     "views": [
      {
       "cell_index": 21
      }
     ]
    },
    "9dba8fca4ba2474e9e6153d51af56994": {
     "views": [
      {
       "cell_index": 13
      }
     ]
    },
    "aa27a3d3608d41df94dbb761c5561cd3": {
     "views": [
      {
       "cell_index": 28
      }
     ]
    }
   },
   "version": "1.2.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}