Human-AI collaboration in large language model-assisted brain MRI differential diagnosis: a usability study

Area of Science:

• Radiology
• Artificial Intelligence
• Medical Diagnostics

Background:

• Large language models (LLMs) show promise in supporting medical diagnosis.
• The specific impact of human-LLM collaboration on brain MRI differential diagnosis accuracy is underexplored.

Purpose of the Study:

• To investigate the effect of human-LLM collaboration on the accuracy and efficiency of brain MRI differential diagnosis.
• To compare LLM-assisted diagnosis with conventional internet search methods.

Main Methods:

• A retrospective study involving 40 challenging brain MRI cases.
• Six radiology residents evaluated cases using conventional search and an LLM-based system in a cross-over design.
• LLM responses were analyzed by expert radiologists to identify benefits and challenges.

Main Results:

• LLM-assisted diagnosis showed significantly higher accuracy (61.4%) than conventional search (46.5%).
• No significant difference was found in interpretation time or reader confidence.
• Key challenges included inaccurate case descriptions, LLM hallucinations, and insufficient contextualization.

Conclusions:

• Human-LLM collaboration can enhance brain MRI differential diagnosis accuracy.
• Addressing challenges in human-LLM interaction is crucial for successful implementation and user acceptance.