Will it run?-A proof of concept for smoke testing decentralized data analytics experiments

Area of Science:

• Health Informatics
• Data Science
• Software Engineering

Background:

• Growing demand for privacy-preserving data analysis in data-driven medicine, exemplified by initiatives like the European Health Data Space (EHDS).
• Distributed Analytics (DA) offers potential for multi-source data analysis but introduces challenges like identifying single points of failure (SPOFs).
• Undetected SPOFs can lead to code malfunctions, research delays, and increased costs.

Purpose of the Study:

• To address the challenge of identifying SPOFs in Distributed Analytics (DA) tasks before execution.
• To introduce and evaluate a novel approach for ensuring the operability of DA analysis code through Smoke Testing.
• To develop an interactive environment for researchers to perform Smoke Tests on DA experiments.

Main Methods:

• Reviewed existing DA platforms to extract six specific Smoke Testing criteria for DA applications.
• Developed the Development Environment for AuTomated and Holistic Smoke Testing of Analysis-Runs (DEATHSTAR) based on these criteria.
• Conducted a user study with 29 participants and applied DEATHSTAR to three real-world use cases.

Main Results:

• The DEATHSTAR environment was assessed for its effectiveness in performing Smoke Tests on DA experiments.
• A user study demonstrated that 96.6% of analyses created and tested using the DEATHSTAR approach terminated successfully without errors.
• The approach effectively identifies potential malfunctions early in the development process.

Conclusions:

• Incorporating Smoke Testing as a fundamental method significantly enhances the reliability of data-driven research using DA.
• DEATHSTAR provides a flexible and adaptable solution for robust and efficient development of DA experiments.
• The developed approach contributes to smoother, more reliable, and cost-effective data-intensive research.