RADICAL-Cybertools: middleware building blocks for NSF's cyberinfrastructure ecosystem

Motivation: Sophisticated and scalable workflows have become essential for advances in computational science. In spite of the many successes of workflow systems, there is an absence of a reasoning framework for end-users to determine which systems to use, when and why. Workflows are increasingly a manifestation of the algorithmic and methodological advances; workflow users and workflow system developers are often the same. Workflow systems must be easily extensible so as to support diverse functionality and the proverbial “last mile customization”. 

We advance the science of workflows and prevent workflow system “vendor lock-in” by formulating a building blocks approach to middleware for workflow systems, grounded on four design principles: self-sufficiency, interoperability, composability and extensibility. A building block has: (i) one or more modules implementing functionalities to operate on a set of explicitly defined entities; and (ii) two well-defined and stable interfaces, one for input and one for output.