Novel Techniques to Assess Predictive Systems and Reduce Their Alarm Burden.

Area of Science:

• Clinical informatics
• Machine learning performance evaluation
• Predictive modeling in healthcare

Background:

• Machine prediction algorithms are often evaluated using standard metrics like precision and recall.
• These classic metrics rely on assumptions that are frequently unmet in clinical workflows.
• This mismatch leads to suboptimal implementation and unmet expectations for predictive tools.

Purpose of the Study:

• To explain the limitations of classic performance metrics in clinical contexts.
• To introduce utility metrics (u-metrics) for more accurate predictor performance assessment.
• To evaluate the effectiveness of a formal snoozing strategy for mitigating false positives.

Main Methods:

• Introduced utility functions to score predictions based on workflow context.
• Developed utility metrics (u-metrics) accounting for temporal relationships and variability.
• Formalized a snoozing strategy to suppress predictions and reduce false positives.

Main Results:

• U-metrics provide a more accurate reflection of real-world predictor costs and benefits.
• The proposed u-metrics significantly improve performance assessment compared to traditional measures.
• Snoozing, when measured by u-metrics, demonstrates measurable performance benefits, particularly for interruptive alarms.

Conclusions:

• Classic performance metrics are inadequate for many clinical machine learning applications.
• Utility metrics (u-metrics) offer a superior method for evaluating predictor performance in real-world clinical workflows.
• Formalized snoozing, assessed via u-metrics, is an effective strategy for enhancing predictor utility and managing alarms.