Explainability in context: calibrating appropriate trust and reliance in artificial intelligence

Predictive artificial intelligence (AI) promises to transform care delivery, enhance patient safety, and improve health outcomes. Realizing these benefits will require careful design, implementation, and monitoring strategies to avoid unintended consequences, including automation bias (i.e., erroneously favoring recommendations from automated systems). Automation bias is particularly concerning due to the variability of AI performance across time and populations, leading to predictions that may be variably incorrect, uncertain, or unfair.

We advocate for an expanded view of explainable AI that uses contextual information to help end users calibrate appropriate levels of trust and reliance. We propose multiple levels of contextualization-model, setting, subpopulation, and patient-that together provide insight for clinicians to evaluate the reliability of individual predictions. This includes information about historical and in-the-moment AI performance, algorithmic fairness, and prediction uncertainty.

We outline an approach to integrate context-based explanations into decision support workflows to aid clinician interpretation without adding cognitive burden.