TutorialAMQP

Hyperflow with AMQP executor tutorial

Installation

This tutorial is based on Ubuntu 14.04 preview release. Here we will assume that everything is installed on one node. All paths will be relative to $HOME.

Prerequisites

• node.js and npm
  • sudo apt-get install nodejs nodejs-dev npm
  • Use nvm if package not available
• Redis
  • sudo apt-get install redis-server
  • Required version >= 2.8
• RabbitMQ
  • sudo apt-get install rabbitmq-server
• Ruby
  • sudo apt-get install ruby2.0 ruby2.0-dev build-essential libxml2-dev libxslt1-dev
  • Required version >= 2.0.0
  • Use rvm if package not available.
• Git
  • sudo apt-get install git

Install Mongage binaries

curl -O http://pegasus.isi.edu/montage/Montage_v3.3_patched_4.tar.gz
tar zxvf Montage_v3.3_patched_4.tar.gz
cd Montage_v3.3_patched_4
make

Install Hyperflow

# We will use development branch
npm install https://github.com/dice-cyfronet/hyperflow/archive/develop.tar.gz

Install executor

curl -O https://dl.dropboxusercontent.com/u/81819/hyperflow-amqp-executor.gem
sudo gem2.0 install --no-ri --no-rdoc hyperflow-amqp-executor.gem # or simply gem if you are using different distribution

Configuration

Redis and RabbitMQ servers

We will stay with defaults for Redis and RabbitMQ, that means no security or authentication. In real deployment you should really take care of that.

AMQP executor

Create executor_config.yml file with the following contents:

amqp_url: amqp://localhost
storage: local
threads: <%= Executor::cpu_count %>

Then you should be able to start executor:

$ hyperflow-amqp-executor executor_config.yml
I, [2014-02-18T17:29:17.069743 #26656]  INFO -- : Starting worker 5eae5b8a-4d73-4d18-b741-126fdcacede2
I, [2014-02-18T17:29:17.075426 #26656]  INFO -- : Running 1 worker threads
I, [2014-02-18T17:29:17.079106 #26656]  INFO -- : Connected to AMQP broker...
D, [2014-02-18T17:29:17.082622 #26656] DEBUG -- : Publishing event executor.ready

Running Montage workflow

• Download example data

  mkdir data
  cd data
  curl -O https://gist.github.com/kfigiela/9075623/raw/dacb862176e9d576c1b23f6a243f9fa318c74bce/bootstrap.sh && chmod +x bootstrap.sh
  ./bootstrap.sh 0.25

• Convert workflow DAX do JSON:

  hflow-convert-dax 0.25/workdir/dag.xml amqpCommand > 0.25/workdir/dag.json

• Set data path

  Edit ~/hyperflow/functions/amqpCommand.config.js to look like this

  var AMQP_URL  = process.env.AMQP_URL ? process.env.AMQP_URL : "amqp://localhost:5672";
  
  exports.amqp_url = AMQP_URL;
  
  exports.options = {
      "storage": "local",
      "workdir": "/home/vagrant/data/0.25/input", // Remember to adjust path here
  }

• Start executor

  export PATH=$PATH:~/Montage_v3.3_patched_4/bin  
  hyperflow-amqp-executor executor_config.yml

• Run the workflow!

  hflow run ~/data/0.25/workdir/dag.json -s

• After minute or two ~/data/0.25/input/shrunken_*.jpg should be created. Note that Hyperflow won't quit after the workflow is finished its work.