Examining reliance patterns on AI advice in medical imaging: a mixed-methods, randomised, crossover experiment

Artificial intelligence (AI) holds significant potential to support diagnostic decision making; however, the collaboration between clinicians and AI systems often does not surpass the performance of unaided humans or standalone AI. This study aimed to examine the effect of AI advice on diagnostic decision making among experts and novices, focusing on the occurrence of AI over-reliance and under-reliance and the potential mitigating effect of explainable AI.

We used a mixed-methods, randomised, crossover experimental design. Participants (50 task experts [radiologists] and 75 novices [non-radiologist physicians and medical trainees]) were randomly assigned in equal proportions to one of six groups, which determined the order in which the three conditions were presented across the three sessions. In each session, participants read 50 head CT scans with or without the presence of an intracranial haemorrhage, under one of three experimental conditions: control, no AI prediction; basic AI advice, AI prediction without annotations; and explainable AI advice, AI prediction with scan annotations. 80% of cases with basic or explainable AI advice received correct advice, and 20% received incorrect advice. The main outcomes were diagnostic performance (the correct identification of intracranial haemorrhage), confidence in the diagnosis, case reading time, and AI advice usefulness ratings. We defined under-reliance as a final decision that did not align with correct AI advice, and over-reliance as a final decision that aligned with incorrect AI advice. All elements of the project were preregistered and are available on the project's Open Science Framework repository.

Data were collected between Feb 16, 2023, and Aug 10, 2023. Both over-reliance on incorrect advice and under-reliance on correct advice occurred. Under-reliance was associated with high uncertainty and, in absolute terms, had a more detrimental effect on diagnostic performance than over-reliance. Correct explainable AI advice reduced under-reliance and improved performance (OR=1·84, p<0·0001, 95% CI 1·49-2·28) and confidence (b=0·15, p<0·0001, 0·09-0·22) compared with no advice, particularly when reviewing more difficult cases with intracranial haemorrhage. Correct explainable AI did not reduce reading time compared with the no-advice condition (b=-1·81, p=0·071, -3·78 to 0·16). Explainable AI was perceived as more useful than basic AI advice (b=0·12, p=0·0029, 0·04-0·20), especially among novices.

The occurrence of both under-reliance and over-reliance highlights the need for efficient counterstrategies beyond classic explainable AI methods to foster appropriate reliance and synergy in AI-supported clinical decision-making.

Volkswagen Foundation, Forum for Interdisciplinary Research, and German Federal Ministry of Education and Research.