Radiologic Decision-Making for Imaging in Pulmonary Embolism: Accuracy and Reliability of Large Language Models-Bing,...

Area of Science:

• Radiology and Medical Imaging
• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems

Background:

• AI chatbots offer potential to improve clinical decision-making and streamline healthcare workflows.
• However, their role in radiologic decision support for specific clinical scenarios like pulmonary embolism (PE) requires further investigation.
• This study assesses the accuracy of prominent Large Language Models (LLMs) for PE imaging recommendations.

Purpose of the Study:

• To evaluate the accuracy and reliability of four leading LLMs (Microsoft Bing, Claude, ChatGPT 3.5, Perplexity) in providing clinical decision support for initial imaging in suspected pulmonary embolism (PE).
• To compare the performance of LLMs across different question formats (open-ended vs. select-all-that-apply).
• To assess the consistency of radiologist agreement on LLM responses.

Main Methods:

• Four variants of PE case scenarios were developed based on the American College of Radiology Appropriateness Criteria.
• Open-ended (OE) and select-all-that-apply (SATA) questions were posed to the LLMs by three radiologists.
• Responses were scored, normalized, and analyzed for accuracy and inter-rater reliability using Intraclass Correlation Coefficient (ICC).

Main Results:

• Perplexity demonstrated the highest accuracy (0.83) for OE questions, while Claude had the lowest (0.58).
• Bing achieved the highest accuracy (0.96) for SATA questions, with Perplexity being the lowest (0.56).
• Overall, OE questions (0.73) scored higher than SATA questions (0.68), with strong radiologist agreement for SATA (ICC=0.875) but poor agreement for OE (ICC=-0.067).

Conclusions:

• Significant variations exist in LLM accuracy for clinical decision support in suspected PE.
• Perplexity and Bing showed strengths in different question formats (OE and SATA, respectively).
• Current LLM inconsistencies necessitate further fine-tuning and careful selection by radiologists for reliable clinical integration.