LABIOS: A Distributed Label-Based I/O System

In the era of data-intensive computing, large-scale applications, in both scientific and the Big Data communities, demonstrate unique, often conflicting, I/O requirements. Further, new storage technologies and system designs often create a heterogeneous complex hardware composition, which is difficult to optimize but ideal for computational storage operations. Existing distributed storage systems are unable to execute different conflicting workloads seamlessly and efficiently. This project introduces a new I/O paradigm that enables a rich feature set addressing the above mentioned I/O challenges. We present the idea of a Label, a novel data representation, and LABIOS, a new, distributed, Label-based I/O system. In LABIOS, all I/O requests are transformed into a Label, a configurable data unit presented as a tuple of an operation and a pointer to its input data. Applications push these data labels into a distributed queue that is served by a distributed I/O scheduler. This queuing system provides the ability to perform asynchronous I/O operations. LABIOS servers, which support tiered storage (e.g., NVMe, SSD, HDD, etc.), execute labels independently. LABIOS elastic architecture allows the server set to be resized based on the I/O traffic within the system. The label abstraction, the decoupled elastic architecture, and the hierarchical storage support will allow LABIOS to provide active storage semantics. LABIOS can significantly boost I/O performance via asynchronous I/O, leverages heterogeneous storage resources, offers storage elasticity, supports software defined storage semantics, and promotes in-situ and in-transit analytics. LABIOS will effectively bridge semantically different storage subsystems to support the undeniable convergence between HPC and Big Data. Building upon the success of the early stages of LABIOS, this project encapsulates a plethora of research activities to enhance, extend, improve, and evaluate this new I/O paradigm to make it practically feasible under the current HPC ecosystem.

This project aims to address several major challenges in the I/O stack. First, it will address the need for storage unification and I/O integration to support diverse scientific domains with varying I/O requirements. Second, it will focus on supporting and leveraging the complex storage hierarchy found in modern systems to accelerate I/O operations. Finally, it will research software-defined storage with in-situ and in-transit computation capabilities to improve the efficiency and productivity of applications running in integrated workflows.

To achieve those, the LABIOS project develops practical solutions, abstractions, and tools to serve the I/O requirements of large-scale data-intensive workloads. The project will enhance the storage stack, benefiting various domains. It will provide scientific applications with a high-performance data movement infrastructure, offer deep learning pipelines access to scientific storage pools, and enable modern applications to interact with a containerized storage substrate. The main goal is to investigate and understand the fundamental properties and tradeoffs of the new I/O paradigm based on data labeling. 

Key objectives include:

• I/O Integration: Seamlessly connect incompatible I/O interfaces to different storage backends while preserving storage semantics.
• I/O Asynchronicity: Enable asynchronous I/O in data processing pipelines, maintaining data durability without explicit control of data flow.
• Storage Dynamic Deployment: Create a flexible storage infrastructure that can be malleable, optimized, and deployed ephemerally through dynamic resource provisioning. 
• Storage Programmability: Achieve programmable storage by offloading data-intensive computations like deduplication, compression, and statistics.