Framework: Software: NSCI: Collaborative Research: Hermes: Extending the HDF Library to Support Intelligent I/O Buffering for Deep Memory and Storage Hierarchy Systems

Modern HPC and distributed systems come equipped with deep memory and storage hierarchies (DMSH). The expert knowledge required to manage these multi-tier storage environments puts their benefits out of reach for most scientists and researchers. In this project, we propose the design and development of Hermes, a new, heterogeneous-aware, multi-tiered, dynamic, and distributed I/O buffering platform which provides: 1) Vertical and horizontal distributed buffering in the DMSH: 2) Selective buffering; 3) Adaptive and dynamic buffering via system and application profiling. We are developing new buffering algorithms and mechanisms that address the challenges of a DMSH ecosystem. This effort will eventually boost HDF5 core technology and facilitate an agile architecture that will allow the evolution of next generation I/O and will address the increasingly challenging scale and complexity of future systems. Hermes software is intended to support new scientific/engineering methodologies and will be carefully designed, implemented, and thoroughly tested.