Comparative Performance of Anthropic Claude and OpenAI GPT Models in Basic Radiological Imaging Tasks

Area of Science:

• Artificial Intelligence
• Radiology
• Medical Imaging

Background:

• Rapid advancements in AI Vision Language Models (VLMs) present opportunities to enhance radiology workflows.
• Evaluating the performance of these VLMs in radiological image interpretation is crucial for potential clinical integration.

Purpose of the Study:

• To assess the accuracy and consistency of publicly available VLMs, specifically Anthropic's Claude and OpenAI's GPT, in basic radiological image interpretation tasks.
• To evaluate model performance across multiple iterations to understand reliability.

Main Methods:

• Utilized subsets from ROCOv2 and MURAv1.1 datasets to test 6 VLMs.
• Inputted system prompts and images into each model thrice, comparing outputs to dataset captions.
• Analyzed accuracy in recognizing modality, anatomy, and detecting fractures, alongside output consistency.

Main Results:

• High accuracy (up to 100%) in modality recognition on ROCOv2.
• Anatomical recognition accuracy ranged from 61% to 85% across models.
• Claude-3.5-Sonnet demonstrated the highest anatomical recognition (57%) and consistency (83% anatomy, 92% fractures) on MURAv1.1; GPT-4o showed the best fracture detection (62%).

Conclusions:

• Current AI VLMs like Claude and GPT lack the necessary accuracy and reliability for clinical radiology integration.
• Ongoing research and development are essential, alongside standardized testing methodologies, to ensure dependable performance of AI in medical imaging.