Minimizing and quantifying uncertainty in AI-informed decisions: Applications in medicine.

Area of Science:

• Artificial Intelligence
• Biomedical Data Analysis
• Statistical Learning

Background:

• AI is crucial for data analysis, but controlling specific error types (e.g., false positives in screening) is challenging.
• Quantifying uncertainty in AI predictions, especially for error control, presents theoretical and practical difficulties.

Purpose of the Study:

• To develop a novel strategy for accurately quantifying AI prediction uncertainty.
• To ensure reliable control over specific error types in AI-driven analyses.
• To address the challenge of trusting AI-based findings in critical applications like biomedical screening.

Main Methods:

• Introduced Multidimensional Informed Generalized Hypothesis Testing (MIGHT), a nonparametric ensemble method.
• Integrated canonical cross-validation and parametric calibration within the MIGHT framework.
• Validated MIGHT's performance through simulations and application to liquid biopsy data (ccfDNA).

Main Results:

• MIGHT accurately quantifies uncertainty and confidence with theoretical guarantees, outperforming typical AI approaches.
• MIGHT demonstrated significantly lower coefficients of variation and often higher sensitivity on ccfDNA data compared to SVM, random forests, and Transformers.
• Found that combining variable sets can decrease sensitivity due to increased noise, highlighting the importance of optimal variable selection.

Conclusions:

• MIGHT provides a trustworthy method for AI-based data analysis, particularly when controlling specific error types is critical.
• The study emphasizes the necessity of quantifying uncertainty and confidence with theoretical guarantees for reliable interpretation of real-world data.
• MIGHT offers a robust solution for biomarker discovery and validation in liquid biopsies, addressing an open question in cancer detection.