Human-AI collaboration in clinical reasoning: a UK replication and interaction analysis

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Clinical Decision Support

Background:

• A prior study indicated large language models (LLMs) outperformed clinicians assisted by the same LLM in diagnostic reasoning.
• The influence of LLMs on clinical decision-making warrants further investigation, particularly in different healthcare settings.

Purpose of the Study:

• To replicate findings on LLM performance versus clinician-LLM collaboration in a UK context.
• To explore how clinician interaction with LLMs impacts diagnostic reasoning performance.
• To identify factors contributing to performance gaps in human-AI collaboration.

Main Methods:

• A within-subjects study involving 22 UK physicians assessing four clinical vignettes.
• Physicians had LLM access via a web application for two cases.
• Analysis included mixed-effects modeling and qualitative coding of LLM interaction logs.

Main Results:

• Physicians with LLM assistance scored significantly lower (21.3 percentage points) than the LLM alone (p<0.001).
• LLM access improved physician performance compared to conventional resources (74.3% vs. 65.7%, p=0.001).
• Qualitative analysis showed only 30% of questions were posed to the LLM, indicating under-utilization.

Conclusions:

• LLM access can enhance diagnostic accuracy, but optimal integration is key.
• Effective human-AI collaboration requires training clinicians to incorporate LLMs into cognitive workflows.
• System design should facilitate seamless integration of LLMs as a default tool.