Eye tracking insights into physician behaviour with safe and unsafe explainable AI recommendations

Area of Science:

• Medical Informatics
• Human-Computer Interaction
• Cognitive Science

Background:

• Explainable AI (XAI) is theorized to aid physicians in high-stakes clinical decision-making by providing context for AI suggestions.
• Objective evaluation of XAI's impact on physician behavior is crucial, as self-reports may not capture true interaction dynamics.

Purpose of the Study:

• To investigate how physicians respond to XAI when presented with safe versus unsafe AI recommendations in a simulated clinical environment.
• To objectively measure the impact of different XAI types on physician attention and decision-making using neurobehavioral methods.

Main Methods:

• Utilized eye-tracking to record overt visual attention of 19 ICU physicians in a clinical simulation suite.
• Assessed prescription decisions and attention patterns before and after revealing safe or unsafe AI recommendations paired with four XAI types.

Main Results:

• Unsafe AI recommendations garnered significantly more physician attention than safe ones.
• No specific XAI type demonstrably increased attention during unsafe AI scenarios, suggesting XAI did not 'rescue' decision-making.
• Physician self-reported usefulness of explanations did not correlate with their actual attention levels.

Conclusions:

• Current XAI approaches may not effectively mitigate risks associated with unsafe AI recommendations in critical care settings.
• Objective measures like eye-tracking are essential for a comprehensive understanding of human-AI interaction, complementing subjective self-reports.