Efficiency Pitfalls of Explainable AI in Clinical Diagnostic and Treatment Human-AI Workflows

Area of Science:

• Artificial Intelligence in Medicine
• Human-Computer Interaction
• Ophthalmology

Background:

• Explainable AI (XAI) aims to build trust but can increase cognitive load and disrupt workflows in high-stakes medical settings.
• Integrating AI transparency features into time-sensitive clinical environments presents unique human-AI interaction challenges.
• Potential for AI system performance impairment due to unintended workflow disruptions.

Purpose of the Study:

• To investigate the impact of AI-provided explanations on ophthalmologists' diagnostic efficiency, accuracy, user perceptions, and workflow integration.
• To explore challenges in human-AI interaction concerning transparency features within time-sensitive clinical diagnostic and treatment workflows.
• To evaluate the trade-offs between AI transparency and operational efficiency in medical practice.

Main Methods:

• A multi-phase, mixed-methods study involving two parts.
• Study 1: A between-subjects experiment (N=32) comparing AI diagnostic support with and without visual explanations for diabetic retinopathy diagnosis.
• Study 2: Qualitative exploration (N=11) using think-aloud protocols and interviews to assess AI experiences in daily treatment workflows.

Main Results:

• Visual explanations did not enhance diagnostic accuracy but increased decision time, negatively impacting efficiency.
• Trends indicated lower perceived usefulness and trust in AI when explanations were provided.
• Qualitative findings revealed explanations were perceived as time-consuming and disruptive, questioning their practical utility in routine cases.

Conclusions:

• A significant trade-off exists between AI transparency and the demand for clinical efficiency.
• AI explanations can hinder efficiency in time-pressured clinical settings, highlighting design challenges for effective human-AI systems.
• Future AI system design should prioritize adaptable, on-demand explanations tailored to user needs for complex cases without disrupting routine tasks.