Related Experiment Video
Updated: Apr 1, 2026
Constructing and Visualizing Models using Mime-based Machine-learning Framework
Published on: July 22, 2025
Predictive performance precision analysis in medicine: identification of low-confidence predictions at patient and
Olivier Lefebvre1, Félix Camirand Lemyre2, Jean-François Ethier3
1Department of Computer Science, Université de Sherbrooke, Sherbrooke, QC J1K 2R1, Canada.
Objectives:
Artificial Intelligence models are increasingly used in health care, yet global performance metrics can mask variations in reliability across individual patients or subgroups with shared attributes, called patient profiles. This study introduces predictive performance precision analysis in medicine (MED3pa), a method that identifies when models are less reliable, allowing clinicians to better assess model limitations.
Materials And Methods:
We propose a framework that estimates predictive confidence using 3 combined approaches: individualized (IPC), aggregated (APC), and mixed predictive confidence (MPC). Individualized predictive confidence estimates confidence for each patient, APC assesses it across profiles, and MPC combines both. We evaluate our method on 4 datasets: 1 simulated, 2 public, and 1 private clinical dataset. Metrics by declaration rate curves show how performance changes when retaining only the most confident predictions, while interpretable decision trees reveal profiles with higher or lower model confidence.
Results:
We demonstrate our method in internal, temporal, and external validation settings, as well as through a clinical example. In internal validation, limiting predictions to the 93% most confident cases improved sensitivity by 14.3% and the area under the receiver operating characteristic curve by 5.1%. In the clinical example, MED3pa identified a patient profile with high misclassification risk, demonstrating its potential for safer deployment.
Discussion:
By identifying low-confidence predictions, our framework improves model reliability in clinical settings. It can be integrated into decision support systems to help clinicians make more informed decisions. Confidence thresholds help balance model performance with the proportion of patients for whom predictions are considered reliable.
Conclusion:
Better leveraging confidence in model predictions could improve reliability and trustworthiness, supporting safer and more effective use in health care.
Insights
This study introduces predictive performance precision analysis in medicine (MED3pa) to identify unreliable AI predictions in healthcare. MED3pa enhances model reliability by flagging low-confidence predictions, improving clinical decision support.
Area of Science:
- Artificial Intelligence in Medicine
- Clinical Decision Support Systems
- Health Informatics
Background:
- Global AI performance metrics can obscure reliability issues for specific patient groups.
- Assessing AI model limitations is crucial for safe healthcare integration.
- Existing methods lack granular analysis of AI reliability across patient profiles.
Purpose of the Study:
- Introduce predictive performance precision analysis in medicine (MED3pa) to quantify AI model reliability.
- Develop a framework to identify and flag AI predictions with lower confidence.
- Enable clinicians to better understand and manage AI model limitations.
Main Methods:
- Propose a framework combining individualized (IPC), aggregated (APC), and mixed predictive confidence (MPC).
- Evaluate the method on simulated, public, and private clinical datasets.
- Utilize declaration rate curves and interpretable decision trees to analyze performance and identify high-risk patient profiles.
Main Results:
- Internal validation showed improved sensitivity (+14.3%) and AUC (+5.1%) when using the 93% most confident predictions.
- MED3pa identified a patient profile with a high risk of misclassification in a clinical example.
- The framework demonstrated effectiveness across internal, temporal, and external validation settings.
Conclusions:
- MED3pa enhances AI model reliability in clinical settings by identifying low-confidence predictions.
- Integrating confidence thresholds balances model performance with patient coverage.
- The framework supports safer and more effective AI deployment in healthcare through improved trustworthiness.
Related Concept Videos
01:13Sensitivity, Specificity, and Predicted Value
Sensitivity is the...
01:15Receiver Operating Characteristic Plot
01:03Prediction Intervals
However, the point estimate is most likely not the exact value of the population parameter, but close to it. After calculating point estimates, we construct interval estimates, called confidence intervals or prediction intervals. This prediction interval comprises a range of values unlike the point estimate and is a better predictor of the observed sample value, y.
01:52Accuracy and Precision
01:52Accuracy and Precision
01:19Interpretation of Confidence Intervals
Confidence intervals have confidence coefficients that are crucial for their interpretation. The most common confidence coefficients are 0.90, 0.95, and 0.99, which can be written as percentages–90%, 95%, and 99%, respectively.
Suppose a person calculates a confidence interval with a confidence coefficient of 0.95. In that case, they can...