Bundle and Pool Architecture for MultiLanguage R

Authors: David Rogers Ian Harvey Tram Truong Huu Kieran Evans Tristan Glatard Ibrahim Kallel Ian Taylor Johan Montagnat Andrew Jones Andrew Harrison

Publish Date: 2013/06/08

Volume: 11, Issue: 3, Pages: 457-480

PDF Link

Abstract

In this paper we leverage the previous work on the SHIWA bundling format and expand on this specification in order to facilitate workflow execution within a multiworkflow environment We introduce a scalable and robust execution pool environment that supports workflows consisting of subworkflows built upon a multitude of different workflow engines and environments and also provide a common workflow representation for seamless connectivity through serialization to workflow bundles We also present a metaworkflow scenario based upon this system Workflow bundles employ the lightweight Open Archives Initiative Object Reuse and Exchange ORE Webbased standard to provide a common format for representing and sharing workflows and the associated metadata required for their execution This generalized bundling approach is already available within five workflow engines and has proven a useful environment for interworkflow experimentation The execution pool facilitates federated access to multiple distributed computing infrastructures supported by the underlying workflow engines subscribed to the pool Workflow bundles are exposed using the eXtensible Messaging and Presence Protocol XMPP which provides the necessary communication backbone to enable multiple workflow engine agents to asynchronously publish and subscribe to bundles in metaworkflow pipelines We present experiments showing the scalability and robustness of the pool execution approach with results showing that overheads remain controlled for up to 150 workflow agents and that agent failures have very limited impact We then demonstrate the applicability of our architecture by describing how a Javabased music analysis workflow can be distributed within such a multiworkflow environment consisting of the Triana and MOTEUR workflow engines

Keywords: