Estimating diagnostic uncertainty in artificial intelligence assisted pathology using conformal prediction.

Area of Science:

• Medical Artificial Intelligence
• Computational Pathology
• Biostatistics

Background:

• Artificial intelligence (AI) systems can produce unreliable predictions when encountering data outside their training distribution.
• Histopathological diagnosis and grading of prostate biopsies are critical for patient management but susceptible to AI errors.
• Ensuring the reliability of AI in medical diagnostics is paramount for patient safety.

Purpose of the Study:

• To demonstrate the utility of conformal prediction for detecting unreliable AI predictions in prostate biopsy analysis.
• To evaluate the impact of conformal prediction on the accuracy and safety of AI-driven histopathological diagnoses.

Main Methods:

• Digitization and analysis of 7788 prostate biopsies for training and 3059 for testing from the STHLM3 diagnostic study.
• Implementation of conformal prediction to identify and flag AI predictions with low confidence.
• Evaluation of AI performance with and without conformal prediction on both internal and external datasets, including small sample validation.

Main Results:

• Conformal prediction reduced cancer diagnosis errors to 0.1% (1 in 794) compared to 2% (14 errors) without it.
• When presented with new data, 22% of predictions were flagged as unreliable, indicating effective detection of out-of-distribution samples.
• In cases of atypical prostate tissue, conformal prediction decreased errors to 2% (3 errors) from 25% (44 errors), flagging 80% as unreliable.

Conclusions:

• Conformal prediction significantly enhances the reliability and safety of AI systems in histopathological diagnosis.
• The method effectively identifies systematic differences in external data, preventing erroneous diagnoses.
• Conformal prediction is a valuable tool for increasing patient safety in AI-assisted medical diagnostics.