Statistical uncertainty quantification to augment clinical decision support: a first implementation in sleep...

Area of Science:

• Medical informatics
• Artificial intelligence in healthcare
• Sleep medicine

Background:

• Machine learning (ML) offers potential in medical pattern recognition, but lacks robust uncertainty assessment.
• Automated medical classification often requires human oversight for accuracy.
• Identifying and addressing classification uncertainty is crucial for reliable AI deployment in clinical settings.

Purpose of the Study:

• To develop and evaluate an uncertainty-based "human in the loop" framework for medical classification.
• To improve the accuracy and efficiency of automated sleep staging using ML.
• To quantify the impact of uncertainty feedback on classification agreement and review time.

Main Methods:

• Implemented an automated single-channel sleep staging system utilizing ML.
• Quantified classification uncertainty using Shannon entropy.
• Developed a "human in the loop" methodology for targeted review of uncertain sleep epochs.
• Evaluated the framework across 20 sleep studies, comparing to gold standard scoring.

Main Results:

• The uncertainty-based feedback methodology significantly improved scoring agreement with the gold standard (average Cohen's Kappa increase of 0.28).
• This approach reduced the overall scoring time by 60% compared to full manual review.
• The system effectively identified uncertain sleep stage classifications for targeted human review.

Conclusions:

• An uncertainty-based clinician-in-the-loop framework enhances medical classification accuracy and confidence.
• This method provides a cost-effective and efficient approach to integrating AI in clinical practice.
• Targeted review of AI-identified uncertainties optimizes resource allocation and improves diagnostic reliability.