Flooding an HPC: Parallelism optimisation in the STRAND project
The STRAND Marsden Fund Project is an interdisciplinary project exploring climate-change flooding related risks to residential property values across space and time, and the related implications for financial stability. One workflow of the STRAND project aims at evaluating the flooding risks for Dunedin properties under various climate change scenarios, architecture choices and the interaction with data uncertainty. This workflow involves generating and aggregating thousands of simulations across many properties.
The embarrassingly parallel nature of the problem made it a good fit for a highperformance computing (HPC) platform. In this talk, we will explore how RTIS (University of Otago) and NeSI partnered to optimise this code. After emphasizing the numerical challenges, we will highlight why and how we combined a workflow management system, Snakemake, and a parallel toolbox, Dask, to distribute tasks efficiently and limit bottlenecks, while keeping the solution portable across two different HPC platforms.
ABOUT THE AUTHORS
Dr Maxime Rio is a data science engineer and data scientist at NeSI and NIWA. He enjoys helping researchers to analyse their data, from visualisation to machine learning and probabilistic modelling.
Dr Murray Cadzow is a Scientific Programmer within Research Teaching IT Support (RTIS) at the University of Otago. Prior to this he spent 11 years researching the genetic basis of gout and related diseases. Murray has been heavily involved in computational literacy and bioinformatic training at the University of Otago - organising Research Bazaar Dunedin and the Otago Bioinformatics Spring School. He is both a Carpentries instructor and instructor trainer. His teaching has focused on delivering digital literacy training to researchers, and the development and support of the local Carpentries community at Otago.
Dr Quyen Nguyen is the STRAND Marsden Fund Project Postdoctoral Fellow (2021–2024) hosted at the School of Surveying, University of Otago. She is the modeller for the STRAND Marsden Fund Project entitled "Should I stay or should I go? Climate-change risks to property values across space and time, and the related implications for financial stability". Dr Nguyen is also working with GNS Science as a Climate Change Economist. Her research interests are in climate finance and data science.
For more information about eResearch NZ / eRangahau Aotearoa, visit:
https://eresearchnz.co.nz/
