Simulating Study Design Choice Effects on Observed Performance of Predictive Patient Monitoring Alarm Algorithms.

Area of Science:

• Biomedical Engineering
• Clinical Informatics
• Health Data Science

Background:

• Evaluating predictive patient monitoring algorithms requires careful consideration of study design.
• Passively collected clinical datasets are often available for retrospective analysis.
• Algorithm performance can be sensitive to definitions of events and true positive alarms.

Purpose of the Study:

• To simulate the impact of varying study design choices on the performance of a predictive algorithm.
• To evaluate the robustness of an early warning score (EWS) for predicting hypotensive events.
• To demonstrate how different alarm and event criteria affect reported algorithm performance.

Main Methods:

• Simulated 432 different combinations of study design choices using retrospective patient monitoring data.
• Varied alarm and event definition criteria, including the look-ahead window for true positive alarms.
• Assessed the impact on performance metrics, such as the area under the receiver-operating characteristic curve (AUC).

Main Results:

• The area under the receiver-operating characteristic curve (AUC) ranged from over 0.8 to below 0.5 based on adjusted criteria.
• Traditional diagnostic system evaluation metrics showed wide modulation due to study design choices.
• The performance of the early warning score was highly sensitive to the selected definitions.

Conclusions:

• Study design choices, particularly alarm and event definitions, critically influence the reported performance of predictive monitoring algorithms.
• A simulation approach using retrospective data is valuable for evaluating algorithm robustness to design variations.
• Careful examination of study design is essential for the reliable development and deployment of new predictive patient monitoring tools.