Lightweight scheduling for the PRAGMA cloud testbed

The Pacific Rim Application and Grid Middleware Assembly (PRAGMA) is a community of research scientists and institutions from around the Pacific Rim that work together to enable scientific expeditions in areas such as biodiversity distribution and lake ecology. PRAGMA's members collaborate on a testbed infrastructure, and over the past 4 years, the technology focus has shifted to cloud and software defined networking as enabling technologies. This paper describes the design of a web‐based cloud scheduler reservation system that enables users to easily run and manage virtual clusters for their science. Based on a client‐server architecture, the cloud scheduler is designed to require minimal installation and management effort for participating PRAGMA testbed sites. The PRAGMA cloud scheduler is built using a web‐based room reservation tool called Booked and leverages several PRAGMA technologies such as pragma_boot, Personal Cloud Controller, and virtual network overlays (eg, IPOP and ViNe). We discuss our implementation of the pilot cloud scheduler and then describe future work.


INTRODUCTION
The Pacific Rim Application and Grid Middleware Assembly (PRAGMA) researchers actively collaborate to enable scientific expeditions in areas of computational chemistry, telescience, biodiversity, and lake ecology. 1  The first phase of the PRAGMA cloud testbed started with 3 sites and focused on the creation of application-specific virtual machines and making them portable to different cloud hosting environments like Rocks, 6 OpenNebula, 7 and Eucalyptus. 8  In late 2014 during the PRAGMA27 workshop, a lightweight scheduler was proposed to coordinate virtual hosts and clusters running on the different cloud deployments and leveraging the above described technologies. The work presented in this paper extends earlier work we presented at a PRAGMA workshop in October 2015. 18 The next section discusses the general requirements of the cloud scheduler, and Section 3 discusses the design options that were considered. The architecture of the cloud scheduler is provided in Section 4 and Section 5 discusses our pilot implementation. Section 6 concludes with our planned future work.

SCHEDULER REQUIREMENTS
The PRAGMA resource providers had the following 3 key requirements for a cloud testbed scheduler to enable multiple users to access the resources at a given time: should first simplify this process by ensuring that pragma_boot gets deployed more broadly and port it to more cloud systems as needed. The scheduler should also provide a simple web interface for users to see the available resources and sites, construct their virtual cluster, and manage their images.
Scale to tens of users: While the PRAGMA community was composed of 19 active institution sites in 2014, the initial number of expected users for the cloud testbed is in the tens of users, not hundreds nor thousands. Therefore, we can prioritize simplicity over scalability and give higher priority to the requirements of low-participation overhead and ease of use.

SCHEDULER DESIGN
In our design discussions, we first looked at some existing solutions that might be adapted for PRAGMA's scheduling needs. Open source batch schedulers like Slurm, 20 TORQUE, 21 or HTCondor, are used to manage space shared resources. There are also other testbeds like Grid' 22 GENI, 23 and PlanetLab 24 that have their own tools for using and managing their infrastructures. In general, we found these tools had different goals and were too heavy weight for PRAGMA. Some tools were better documented than others but all had a fairly high learning curve and would have been non-trivial to adapt for PRAGMA's cloud testbed, which had a strong application and virtual cluster focus.
We next searched for a simpler approach to scheduling and found a number of open source web-based calendar reservation systems that seemed promising. These systems were designed to manage meeting room reservations, hotel room reservations, or equipment rentals.
Their interfaces would be more familiar to users in their everyday lives and thus well suited to the different types of PRAGMA users. In addition, several of these tools were significantly less complex than the batch and testbed scheduling tools above. Later, we found a web-based calendar reservation system called GridPrems 25 for Grid'5000 indicating a simpler interface was sometimes needed even for their users.
Some of the calendar reservation systems we looked at like Reservations, 26 Merci, 27 and Booking_calendar 28 were plugins for Drupal 29 or Wordpress. 30 However, the majority of tools were PHP based with a relational database backend like the Meeting Room Booking System, 31 Booked, 32 and Classroombookings. 33 For each system, we evaluated (1) how easy it would be to manage resources, reservations, and users as well as to add new parameters and features; (2) how intuitive the GUI interface was with respect to menus and navigation and if it had a clean, modern, and uncluttered look; (3) how easy it was to install and setup a prototype instance. One deficiency that all systems had was that they only allowed 1 reservation per resource.
Booked 32 from Twinkle Toes software seemed the best suited to our needs compared to all the calendar reservation systems we evaluated.
It was easy to setup, had a good look and feel, and was customizable.
It also had some nice additional features like usage reporting, support for different time zones for each user, a REST API interface, LDAP and Active Directory support, fine grained roles and permissions management, and user and group quotas. After working with Booked during our pilot setup, we did find some weaknesses. First, the documentation was very sparse and any needed PHP changes were time consuming due to its heavily object-oriented structure. For example, we had to go through many source code files to find the right place to change or add functionality. Second, the reservation frequency of the event is currently limited to 1 hour. Based on today's PRAGMA needs, this is sufficient because the time it takes to start up the virtual cluster is in the order of tens of minutes and the user reservation is expected to be in the order of days.
In the future when the virtual cluster startup time shortens to minutes, the reservation frequency granularity can be optimized to avoid a potential of the underutilized resources.

SCHEDULER ARCHITECTURE
One key requirement for our cloud scheduler was to enable users to create and manage virtual clusters on the PRAGMA cloud testbed with minimal effort. The basic workflow and steps for how a user would start a virtual cluster are summarized below: 3. The user selects the desired resources, the timeframe, virtual cluster image, and configuration details and submits them to the Cloud scheduler. Once the reservation is verified and confirmed, the user will get an email confirmation. In the event 2 users try to reserve the same resource at the same time, for example, 1 user has taken the resources while the second user was filling in the reservation fields; the second user will get a failure notification after hitting the submit button.
4. When the reservation is ready to be activated, the cloud scheduler will use PCC and pragma_boot to launch the virtual cluster, automatically configure the private and public network configuration on the virtual cluster nodes, and email the user when the reconfigured cluster is ready. Figure 1 shows the above steps in a use case sequence diagram where PRAGMA Booking refers to our customized Booked instance.
The PCC will monitor the health of the virtual cluster and when the reservation is close to expiring, will notify the user via email. The user then has an option to extend the reservation if resources are available. Otherwise, when the time expires or the user chooses to end the reservation, the virtual cluster will be shut down.

SCHEDULER PILOT
In our pilot implementation of the cloud scheduler, we made a number of simplifying assumptions to the design.
Virtual cluster images are already available at each site: For our pilot, we worked with a small number of virtual cluster images and preinstalled them to each cloud system so they were readily accessible by pragma_boot. We had the following 3 images: (1) a basic SGE Rocks virtual cluster; (2) an AutoDock virtual cluster; (3) a Lifemapper 35 compute virtual cluster. • Added the ability to make multiple reservations on a resource per time slot to allow more than 1 virtual cluster to run simultaneously on a site. This was the most extensive change we made to Booked.
Fortunately, no changes were needed to the backend database, but it required several changes to the GUI code to allow an additional reservation to be made on a resource and to display multiple reservations per time slot in the calendar views. In total, enhancements were made to seven PHP files and 2 template files.
• Added a custom field for a public SSH key to the user profile. When PCC launches a virtual cluster, it pulls the public SSH key from the user's profile and gives it to pragma_boot so that the user can login as root once the virtual cluster is launched. The ability to use a root login is not required for all but can be needed in some case if users want to do system modifications. A user with a root login has a full ownership of the virtual cluster.
• Added custom fields for CPU count, amount of memory per host, and virtual cluster image name to the reservation specification.
• Added custom fields for CPU count, amount of memory per host, site hostname, and ENT capability (enabled and disabled) to the resource specification.
• Added a numeric count as a custom field type. Booked only provided text matching on search so if a host had 16 CPUs available and the user requested 8 CPUs, the search would fail per "16" != "8" inequality. By adding count as a custom field type, the search would do a numeric comparison on the values and the search would succeed for "8 <= 16".
• Added virtual cluster statuses: "Starting", "Running", and "Stopping". Each virtual cluster status is shown as a different color in the calendar view. "Starting" means the virtual cluster has been accepted by the scheduler and is being started up on the local cloud system, "Running" means that the virtual cluster is available for the user to log into, and "Stopping" means the virtual cluster is scheduled for shutdown and is no longer available for log in.
• Added the ability to retrieve and set the reservation status from the Booked REST API.
• Added the PRAGMA logo to the header. In the pilot implementation of the cloud scheduler we made a lot of simplifying assumptions, and our future work will address these. Currently, the virtual cluster startup latency from the point in time of its reservation to its availability to the user has not been analyzed in detail but we observed it to be on the order of tens of minutes. Multiple factors influence this latency, such as the underlying virtualization manager and load on the physical system where the virtual cluster is instantiated. There is additional latency if the cluster image is not available on the site as it will need to be downloaded from a remote repository and processed to make it available on the site. While this applies only to the first virtual cluster instantiation, it can add a considerable time latency depending on the cluster image size and the network speed from the site to the remote repository.
Once the cloud scheduler is deployed to more sites, we can gather statistics on its usability and user experiences with virtual clusters startup latencies in addition to its applicability to different scenarios. We will also work on integrating IPOP and PRAGMA-ENT into the cloud scheduler. Next, we will rework the PCC software and integrate HTCondor so it is used in personal mode allowing us to keep a light footprint at each of the PRAGMA sites. Finally, we will investigate using the remote repository option in pragma_boot to manage application-specific virtual cluster images and staging them to the sites.
We will package and document the resulting software so that new users can quickly start using the PRAGMA cloud testbed as an experimental service to test new science and infrastructure tools.