ChatGPT, Claude Sonnet, and Grok Display Similarly Low Rates of Accuracy in Identifying Image-Based Orthopaedic...

Area of Science:

• Orthopaedic imaging analysis
• Artificial intelligence in medicine
• Diagnostic accuracy assessment

Background:

• The increasing popularity of AI necessitates understanding its diagnostic capabilities and limitations.
• Radiologic imaging is crucial for diagnosing sports-related pathologies.

Purpose of the Study:

• To evaluate the accuracy of AI in identifying common sports injuries from radiologic images.
• To compare the performance of ChatGPT 4.0, Grok 2, and Claude 3.5 Sonnet.

Main Methods:

• Five common sports pathologies (ACL tears, PCL tears, meniscal tears, chondral pathologies, rotator cuff tears) were selected.
• Fifty images per pathology were collected from imaging databases (radiography, CT, MRI), including normal images.
• Receiver operator characteristic curves and area under the curve (AUC) values were calculated to assess AI accuracy.

Main Results:

• ChatGPT 4.0, Grok 2, and Claude 3.5 Sonnet identified pathologies in 23.6%, 15.7%, and 17.1% of diseased images, respectively.
• AUC values were low (ChatGPT: 0.21, Grok: 0.16, Claude: 0.15), indicating poor performance.
• No significant differences in accuracy were observed between the AI platforms overall or for specific pathologies.

Conclusions:

• Current generative AI models demonstrate significant limitations in accurately diagnosing sports-related pathologies from radiologic images.
• The accuracy rates and AUC values suggest these AI tools are not yet suitable for clinical use in orthopaedic image-based diagnosis.
• Further development is required before AI can be reliably integrated into diagnostic workflows for musculoskeletal conditions.