Resource planning on the Cloud: exploring the scalability spectrum

Cloud computing resources have become the informatics backbone for scalable, accessible, customizable, and secure computing with bioinformatics continuing to benefit from this computational model. What started as a handful of applications that were ported to the Cloud has geared up to creation of Virtual Laboratories and Cloud Pilot projects funded by national funding agencies. Today, a typical end-user scenario for the cloud is to acquire a set of virtual machines from a cloud provider with pre-installed software and perform the needed data analysis. In the process, the user needs to make cost-effective decisions about what resources to acquire and how many. These decisions have a direct impact on the outcome of the analysis because with insufficient resources it may be impossible to complete the analysis or it may take extra time. Excessive resources waste project funds or merit allocation credits and can cause resource contention on academic clouds.

To shine some light on this topic, we performed a number of experiments with the Galaxy CloudMan project to explore the tradeoffs among resource types and sizes across the Amazon Web Services infrastructure. Using published next generation sequencing data we identified resource requirements, limits on resource classes, and observed actual resource utilization for RNA-seq and chIP-seq pipelines. These results can be used help users gauge what resources to use when using cloud machinery. They can also be used by academic cloud infrastructure projects to determine what type of underlying infrastructure is needed by users. In this talk, we will detail our findings.