Understanding Uncertainty in Large Language Model Predictions of Early Death in Critically Ill Patients: A Conformal...

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Decision Support

Background:

• Early prediction of in-hospital death is challenging due to limited structured data.
• Unstructured clinical notes are an underutilized resource for real-time risk stratification.
• Large language models (LLMs) offer potential but lack patient-level uncertainty quantification.

Purpose of the Study:

• To evaluate the effectiveness and confidence of LLMs (GPT-4o) in predicting in-hospital death probability.
• To quantify the uncertainty of LLM predictions using unstructured clinical notes.
• To assess the applicability of LLM predictions for clinical decision-making.

Main Methods:

• Applied conformal prediction (CP) to quantify uncertainty of GPT-4o zero-shot predictions.
• Utilized concatenated clinical notes from the first 24 hours of ICU admission (MIMIC-III).
• Focused on patients with acute kidney failure admitted via the emergency department (ED).

Main Results:

• GPT-4o achieved a recall of 0.93 for in-hospital death.
• Conformal prediction framework provided overall empirical coverage of 90.4%, exceeding the 90% target.
• Class-specific coverage was imbalanced (99.7% for death, 81.1% for survival).

Conclusions:

• LLM outputs can be overconfident, especially for incorrect predictions.
• Conformal prediction is a promising approach for quantifying and calibrating LLM uncertainty.
• Integrating CP enhances the potential applicability of LLMs for clinical decision-making.