Let XAI generate reliability metadata, not medical explanations

Area of Science:

• Medical Artificial Intelligence
• Clinical Decision Support Systems

Background:

• Artificial intelligence (AI) is increasingly integrated into healthcare.
• Regulatory demands and clinical practice necessitate explainable AI (XAI).
• Conventional XAI methods often fail to align with clinical decision-making processes.

Purpose of the Study:

• To challenge the conventional focus of XAI on post-hoc explanations.
• To propose "reliability metadata" as a more effective approach for calibrating physician trust in AI.
• To advocate for AI designs that support adaptive reliance and mitigate automation bias.

Main Methods:

• Critiquing current XAI approaches in the context of clinical cognition.
• Proposing a shift from static explanations to actionable cues.
• Introducing the concept of "reliability metadata" including marginal and instance-specific indicators.

Main Results:

• Conventional XAI's focus on post-hoc explanations is misaligned with clinical needs.
• Reliability metadata, including confidence scores and out-of-distribution alerts, can enhance trust and safety.
• Reframing XAI as "eXtended and eXplorable AI" promotes interaction and uncertainty transparency.

Conclusions:

• Physicians require actionable reliability metadata, not just explanations, for effective AI integration.
• AI design should prioritize transparency, interaction, and clinical relevance to support physician decision-making.
• This approach fosters safer and more effective use of AI in medical practice.