Benchmarking the diagnostic performance of open source LLMs in 1933 Eurorad case reports

Area of Science:

• Artificial Intelligence in Medicine
• Radiology and Medical Imaging

Background:

• Large language models (LLMs) offer novel applications in medical diagnostics.
• Open-source LLMs present potential advantages in cost and access compared to proprietary models.
• Evaluating LLM performance in real-world clinical scenarios is crucial.

Purpose of the Study:

• To assess the diagnostic performance of open-source and closed-source LLMs in radiological differential diagnosis.
• To compare the accuracy of various LLMs using a large, public dataset and a private dataset.
• To determine the potential of open-source LLMs as clinical decision support tools in radiology.

Main Methods:

• Evaluated 15 open-source LLMs and GPT-4o on 1,933 cases from the Eurorad library.
• LLMs generated differential diagnoses based on clinical history and imaging findings.
• Tested generalizability on 60 non-public brain MRI cases from a tertiary hospital.

Main Results:

• GPT-4o exhibited superior diagnostic performance across both datasets.
• Llama-3-70B demonstrated strong performance, closely following GPT-4o.
• Open-source LLMs are rapidly improving and narrowing the performance gap with proprietary models.

Conclusions:

• Open-source LLMs are becoming increasingly capable for radiological differential diagnosis.
• These models show significant potential as decision support tools in challenging radiological cases.
• Continued development of open-source LLMs could enhance diagnostic workflows in radiology.