Simulating Study Design Choice Effects on Observed Performance of Predictive Patient Monitoring Alarm Algorithms

There are multiple study design choices to be selected in order to perform evaluations of predictive patient monitoring algorithms related to the event and true positive alarm definitions (e.g., how far ahead of the event is a true positive alarm). Often, passively collected patient monitoring datasets from clinical environments are available to perform these types of studies, so that the effects of different study design choices can be simulated to evaluate the robustness of an algorithm to those choices. Here, we simulate the effects of varying alarm and event definition criteria on the reported performance of the early warning score to predict hypotensive events. A total of 432 combinations of study design choices were simulated. Area under the receiver-operating characteristic curve varied from greater than 0.8 to less than 0.5 by adjusting alarm and event definition criteria. Traditional metrics for evaluating diagnostic systems were modulated across a wide range by adjusting study design choices for a predictive algorithm using a patient monitoring dataset. This highlights the importance of examining study design choices for new predictive patient monitoring algorithms and presents an approach to simulate different study designs with retrospective patient monitoring data as part of a robustness evaluation.