SFB 1404/2: FONDA - Foundations of Workflows for Large-Scale Scientific Data Analysis

Abstract

Scientific discoveries in the natural sciences rely on the computational analysis of large data sets, which are carried out by complex data analysis workflows (DAWs) executed on a distributed infrastructure. Most research in DAWs focuses on techniques for minimizing their runtime on a specific infrastructure, which leads to solutions that are difficult to maintain and dependent on the involvement of highly specialized and scarce data engineers. However, in most data science projects, runtime is not the decisive factor; instead, it is its development time. FONDA set out in 2020 to address this long-lasting and increasingly pressing problem. Our overarching research goal is to research languages, technologies, and algorithms to increase human productivity when designing, maintaining, or reusing DAWs for large-scale scientific data analysis. Within its first funding period, FONDA focused on three specific properties of DAWs that are directly linked to human productivity, namely portability, adaptability, and dependability. FONDA achieved groundbreaking results in these regards, such as improved portability through flexible interfaces between infrastructure components, improved adaptability via intelligent scheduling, and improved dependability through contract-driven DAW development. In its second phase, FONDA will further develop its research topics by lifting three restrictions we imposed on ourselves in phase I. First, we break the assumption that DAWs are executed in a single data center hosting all necessary data and will study multi-site DAWs, i.e., DAWs whose sub-workflows are executed in different data centers. Second, we extend our scope in terms of the DAW lifecycle by addressing usability of DAW systems, i.e., empirical investigations of hu-man-computing interfaces and a systematic approach to DAW design. Third, we generalize from single workflows to workflow reuse by researching the technical sustainability of DAWs. Furthermore, as human productivity in data analysis is increasingly threatened by excessive energy costs, we take improvements to environmental sustainability in focus. Besides its scientific results, FONDA’s first phase also excelled in several overachieving topics. With the recent founding of the new HPC@HU service, it had a long-lasting structural impact on the speaker university. The recognition of its highly important research topic at the interface be-tween computer science and the natural sciences is reflected by many recent appointments in the region, which allowed a perfectly matching extension of our PI group. We are proud to have achieved an outstanding high percentage of female PhD students (38%), and we are looking forward to the new edited book on “Workflows for Large-Scale Scientific Data Analysis”, for which more than 100 authors from 15 countries have confirmed contributions and that will ap-pear in summer 2024 in the newly created Open Access publisher BerlinUP.