The human factor in explainable artificial intelligence: clinician variability in trust, reliance, and performance

Area of Science:

• Medical Imaging
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Explainable Artificial Intelligence (XAI) is crucial for high-risk fields like healthcare to build user trust.
• Current evaluations often use automated metrics, neglecting crucial user-centered assessments.

Purpose of the Study:

• To evaluate the impact of a prototype-based XAI model on clinician trust, reliance, and performance in image-based gestational age estimation.
• To introduce and assess a novel metric for appropriate reliance on AI in clinical settings.

Main Methods:

• A prototype-based XAI model was adapted for gestational age estimation from medical images.
• Ten sonographers participated in a three-stage reader study to assess the XAI model's effects.
• Clinician performance, trust, reliance, and confidence were measured, including a new metric for appropriate reliance.

Main Results:

• The XAI model reduced the mean absolute error (MAE) in gestational age estimates from 23.5 to 15.7 days.
• Explanations provided by the XAI model led to a further, though not statistically significant, reduction in MAE to 14.3 days.
• Participant responses to explanations varied, with some performing worse; confidence increased, but trust and reliance remained unaffected.

Conclusions:

• Explainable Artificial Intelligence (XAI) deployment in clinical settings requires careful consideration of user variability and its impact on performance.
• Automated metrics are insufficient; human studies are essential to understand the nuanced effects of XAI on clinician behavior and decision-making.
• The findings underscore potential challenges in achieving desired trust and reliance outcomes with current XAI approaches in medical applications.