Pub/Sub for task-chaining and external-communications

[daigotanaka]

Currently, handoff tasks are triggered by CRON schedules.
I'm thinking to add a Pub/Sub feature. This direction makes a lot of sense in terms of

1. Chaining multiple tasks with dependency
2. Fanning out and scaling tasks
3. Notification and monitoring

...all along with the serverless philosophy of handoff.

• Pub/Sub topic could be outside of handoff configured ones.
• A handoff task can be triggered when a message is published to the topic.
• A handoff task can publish a message when it completes or fails.

Example use-case: ELT orchestration

1. CRON triggers the Extraction of cloud application data and Load on a data warehouse (Task A).
2. Task A publishes the completion message to the topic.
3. Task B subscribes to the topic and kicks off a dbt job in the data warehouse to transform the data.

AWS implementation ideas

As for AWS implementation, which is the only handoff platform implementation for now,

• Use AWS SNS for Pub/Sub implementation
• Use FIFO to prioritize the message deduplication than the message through-put because of the nature of the data pipeline process.
• handoff's task Cloudformation template to create a topic.

Then maybe we can define in project.yml something like:

- publishes
  - on_success:
    - topic: <topic_identifier>
  - on_failure:
    - topic: <topic_identifier>

We can also set the default <topic_identifier> to be the one created with the task Cloudformation template.

On the subscription side, Fargate is not well connected to SNS as of today. We need to insert Lambda or SQS to trigger Fargate. This gets a little ugly, but that's what handoff is for. We can provide a simple and clean way of defining what we want to do and handoff handles the ugly AWS wiring (and easy transition to other cloud providers (Google Cloud Platform, Azure) in the future implementation.)