Solving the explainable AI conundrum by bridging clinicians' needs and developers' goals.

Area of Science:

• Healthcare Informatics
• Artificial Intelligence
• Human-Computer Interaction

Background:

• Explainable artificial intelligence (XAI) is crucial for AI/ML adoption in healthcare.
• Understanding developer and clinician perspectives on XAI is limited.
• Conflicting goals and requirements can hinder XAI implementation.

Purpose of the Study:

• To investigate the differing mental models of developers and clinicians regarding XAI.
• To identify conflicting goals and requirements in XAI co-design for healthcare.
• To propose design solutions for effective XAI in clinical decision support systems.

Main Methods:

• Longitudinal multi-method study.
• Co-design of an XAI solution for a clinical decision support system.
• Involved 112 developers and clinicians.

Main Results:

• Identified three key differences in developer vs. clinician mental models of XAI.
• Differences include opposing goals (interpretability vs. plausibility), sources of truth (data vs. patient), and knowledge exploration vs. exploitation.
• Highlighted the need to reconcile these differing perspectives.

Conclusions:

• Reconciling developer and clinician XAI mental models is vital for healthcare AI.
• Proposed design solutions: causal inference, personalized explanations, and ambidexterity.
• Emphasized considering both stakeholder perspectives for effective XAI in healthcare.