Dapp challenge - decentralised model exploration and parameter tuning with OpenMOLE

.gitignore

@@ -0,0 +1,32 @@
+# Logs
+logs
+*.log
+npm-debug.log*
+
+# Runtime data
+pids
+*.pid
+*.seed
+
+# Coverage directory used by tools like istanbul
+coverage
+
+# node-waf configuration
+.lock-wscript
+
+# Dependency directory
+node_modules
+
+# Compiled JS directory
+/dist/*
+!/dist/iexec.js
+
+# Optional npm cache directory
+.npm
+
+# Optional REPL history
+.node_repl_history
+
+package-lock.json
+wallet.json
+account.json

LICENSE

@@ -0,0 +1,21 @@
+The MIT License
+
+Copyright (c) 2017 iExec Blockchain Tech, https://iex.ec
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

README.md

@@ -1,13 +1,103 @@
-# iexec dapps samples
+# Submission to DApp challenge: the OpenMOLE scientific platform, decentralised model exploration and parameter tuning on the iEx.ec network
 
-This is the registry for sample iexec dapps, used by the iexec-sdk cli.
+## Idea proposal
+[OpenMOLE](https://www.openmole.org) is a generic scientific exploration platform with a strong support for High-Throughput Computing environments. Its principal purpose is to provide advanced **model exploration methods and parameter tuning** based on distributed genetic algorithms. OpenMOLE exposes a workflow formalism to describe the experiments and as such, can serve as a generic description language for processing pipelines.
 
-Each branch name of this repo can be used as an argument to iexec init command.
+The power of OpenMOLE is further expressed in its ability to **automatically introduce parallelism to the underlying applications**. We exploit the intrinsic parallel aspect of data parallel problems without having to modify the target application’s source code. Studied applications are viewed as black-boxes by the platform, which ensures the required inputs and outputs travel to the remote execution nodes.
 
-ex:
+The typical uses of OpenMOLE are **model calibration, model exploration, machine learning, optimisation and data processing**. End-users can specify the execution environment for each stage of the experiment. OpenMOLE currently supports clusters (Slurm, HT Condor, PBS, …) and grids such as the **European Grid Infrastructure (EGI)**.
 
-```bash
-iexec init
-iexec init factorial
-iexec init echo
-```
+This project proposal introduces an OpenMOLE DApp leveraging the computing resources of the iEx.ec network to decentralise the execution of OpenMOLE’s **scientific use-cases**.
+
+## Background
+Over the last 10 years, we have developed an expertise in distributed computing through our respective research interests requiring large quantities of processing power. These efforts are gathered in the scientific platform OpenMOLE, an open source software leveraging **distributed computing to explore large parameter spaces** using state of the art methods.
+OpenMOLE has been used to calibrate simulation models through the use of large scale distributed computing environments (http://journals.sagepub.com/doi/10.1068/b130064p). Our current setup enables users to delegate their compute-intensive tasks to local clusters (*e.g.* via Slurm, HT Condor, *etc.*) as well the European Grid Infrastructure (EGI).
+
+Working with heterogeneous computing environments such as the EGI has led us to build advanced task (re-)submission, file transfers and automatic results retrieval mechanism in OpenMOLE.
+As strong advocates of **reproducible science**, we recently published a method to reproduce **[neuroimaging experiments at scale](http://journal.frontiersin.org/article/10.3389/fninf.2017.00021/full#)** with the OpenMOLE platform. It focuses on providing a standalone solution mimicking the packaging features provided by traditional container solutions such as Docker and Singularity, without relying on a pre-installed piece of software on the execution host. This work allowed us to develop an **expertise to package complex scientific pipelines**. Further research building on this initial achievement has resulted in techniques to manipulate docker images on restricted environments where no docker installation is available (*to be published*, an early description is available [in the documentation](https://next.openmole.org/Container.html)).
+
+Another aspect of our research embedded in OpenMOLE is the conception of Evolutionary/Genetic Algorithms (EA/GA) based optimisation methods. These methods provide support for **Parameter exploration**. As models become more and more complex, they display a large number of parameters. Most of the time, finding the best parameter settings rely on the expert's knowledge. However, this might not be the optimal set of parameters, or an expert choice can hide some aspects of the model. Using scientific exploration platforms like OpenMOLE is a way for scientists to understand their models better. In some cases, it's even the only way to study a model.
+
+Beyond parameter tuning, we have proposed a novel method to explore [all possible behaviour of a given model](http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0138212). While OpenMOLE roots in the modelling and simulation community, it has proved to be generic enough to be employed to address problems in various scientific fields ranging from [geography and social sciences](http://jasss.soc.surrey.ac.uk/18/4/9.html) to [biomedical image analysis](https://hal.inria.fr/hal-01099220/).
+
+Today we aim to bring the optimisation and parameter space exploration methods at the heart of the OpenMOLE ecosystem to the iEx.ec platform. We believe our joint expertise in modelling and simulation, optimisation methods and distributed computing places us in a unique position to build the **first scientific DApp on the iEx.ec DApp store**.
+
+## Roadmap
+We expect to provide a first prototype of our solution for Q3 2018. Further refinements will be made based on the initial feedback in order to release a beta version in Q4 2018.
+
+The steps taken will be as follows:
+
+1. Preliminary version running OpenMOLE on the iEx.ec worker nodes and fetching a predefined execution context: a simple task from the OpenMOLE marketplace estimating the value of Pi using a Monte Carlo method. At this stage, we only consider the OpenMOLE components running on the worker nodes. The user will not interact with the OpenMOLE web platform, but will instead directly use the iEx.ec SDK to vary the input parameters of the application. **This is the state at the time of submission**.
+
+2. Integrate the iEx.ec SDK in OpenMOLE. An initial version will embed the iEx.ec NodeJS application in OpenMOLE and call it as an external process to submit new jobs to the network. We will discuss with the iEx.ec team the possibility to see the SDK translated to other languages in the future. The job’s configuration as JSON files will be automatically generated by the platform, and passed to the `iexec-submit` command.
+This **second milestone** is reusing the Pi simulation, and will automatically create a test wallet and fill it with test ETH/RLC.
+
+3. Manage the iEx.ec wallet from OpenMOLE’s authentications mechanism. The user should be able to retrieve his iEx.ec wallet from the OpenMOLE interface and define the same limits as from the SDK regarding his RLC spending cap. The user’s private key will never travel away from his machine and will be stored encrypted in OpenMOLE’s preferences. This will enable **transparent task submissions** to the iEx.ec network from the OpenMOLE web interface.
+
+4. Our **fourth milestone (M4)** will start tackling **input data management**. Up until now, a single application was considered and manually serialised to the iEx.ec network along with the OpenMOLE runtime. Milestone **M4** will bring the possibility to serialise tasks described from the OpenMOLE web interface, and provide them to the iEx.ec platform upon submission. We will reuse mechanisms such as `dirinuri` to transfer the data or any better scheme suggested by the iEx.ec team. Upon completion of **M4**, it will be possible to delegate the execution of generic tasks, involving no or low data transfers, to the iEx.ec network.
+
+
+5. Milestone **M5** will strengthen the data management component of our solution. We expect this to be in line with iEx.ec’s developments towards V3 of the network. OpenMOLE should be able to upload the user’s data to a decentralised storage managed either by iEx.ec or the OpenMOLE project. An initial version will manually spawn and maintain *IPFS* nodes (or equivalent), before switching to iEx.ec’s solution.
+
+6. With larger data managed, the **OpenMOLE DApp** will be ready for deployment on the App Store. The final milestone (**M6**) will define a monetisation scheme for OpenMOLE in accordance with other practices on the iEx.ec network and the input from the team. It should be the opportunity to involve a larger community of beta-testers, and incorporate their useful feedback.
+
+
+## Estimated timeline
+
+- M1: February 2018
+- M2: March 2018
+- M3: April 2018
+- M4: June 2018
+- M5: Q3 2018
+- M6: Q4 2018
+
+
+## Diagrams
+
+### Component diagram
+
+Below is a component diagram giving a high level view of the elements interacting in our proposal.
+
+![component_diagram](images/component_diagram.png)
+
+### Sequence diagram
+
+To visually express the flow which the desired program will have, the sequence diagram below sketches a scenario involving two iEx.ec worker nodes. The storage location is a modular
+component and will be further investigated in the implementation stage.
+
+
+![sequence_diagram](images/sequence_diagram.png)
+
+## Benefits for iEx.ec
+
+Our submission will bring multiple valuable assets to the iEx.ec network.
+
+- A **scientific DApp** introducing the network to scientists not yet familiar with web3 technologies.
+- An **entry point** for other applications lacking the expertise to bring their software to the iEx.ec network. They can rely on OpenMOLE to delegate their workload to the iEx.ec network.
+- An extensive roadmap that can lead to a **generic support of Map/Reduce applications**, and refining the task and data management specifications.
+
+
+## Team
+
+- Romain Reuillon
+CNRS Researcher at Géographie-cité and ISC-PIF. Scientific Manager of the OpenMOLE platform leader of the development for Scientific simulation and computing, he developed both MGO and GridScale.
+
+- Mathieu Leclaire
+CNRS Researcher Engineer at ISC-PIF (UPS 3611) and Géographie-cité (UMR 8504). Developer of the OpenMOLE Platform, he developed both the Scaladget library and ScalaWUI skeleton.
+
+- [Jonathan Passerat-Palmbach](https://jopasser.at)
+Research associate at Imperial College London, in the BioMedIA group. Developer of the OpenMOLE Platform, he co-developed GridScale.
+
+- Karow Maruf
+MEng student at Imperial College London.
+
+
+## Bonus
+
+**Don't take our words for granted!** You can already estimate the value of Pi on the iEx.ec test network with our [very early implementation (M0)](apps/README.md).
+
+## References
+
+- [1] [OpenMOLE website](https://www.openmole.org)
+- [2] [OpenMOLE code base and related projects](https://github.com/openmole)
+- [3] [The academic papers mentioned in this document](https://openmole.org/Communications.html)

apps/OpenMOLEPi

@@ -0,0 +1,24 @@
+#!/bin/bash
+
+BASE_DIR="$(pwd)"
+
+wget https://openmole.org/openmole.tar.gz
+tar zxf openmole.tar.gz
+rm openmole.tar.gz
+
+cd openmole || exit 1
+
+mkdir -p java_om && tar zxf runtime/jvm-x64.tar.gz -C java_om
+JAVA_HOME="$(pwd)/java_om/jre"
+PATH="${JAVA_HOME}/bin:${PATH}"
+
+export JAVA_HOME PATH
+java -version
+
+wget https://raw.githubusercontent.com/openmole/openmole-market/7-dev/pi/Pi.oms
+pi_avg=$(./openmole --password "" --script ./Pi.oms | tail -n1 | sed -e 's/.*\([0-9]\+\.[0-9]\+\).*/\1/g')
+
+echo "${pi_avg}"
+
+cd "${BASE_DIR}" && rm -rf openmole
+

apps/README.md

@@ -0,0 +1,17 @@
+## Estimation of Pi via a Monte Carlo method
+
+This workflow exposes how to compute an estimation of Pi usign a Monte-Carlo estimation of Pi. The computation task is a Scala task using a self contained code. The workflow computes 100 independant executions of this task using randomly generated seeds for the random number generator. It produces 100 independant realisation of the Monte-Caro estimation. Then the workflow gathers them in a vector and computes an average on the values of the vector. Finally the result is displayed on the standard output.
+
+[Original source](https://github.com/openmole/openmole-market/tree/7-dev/pi)
+
+## Deployment
+
+```shell
+iexec wallet create
+iexec wallet getETH
+iexec wallet getRLC
+iexec deploy # if you've modified the app
+iexec submit
+iexec result <txID> --save # will return an approximate value of Pi in stdout.txt
+```
+