Evaluating the Performance of a ChatGPT Model in Rheumatology Exams

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Rheumatology Diagnostics

Background:

• Large language models (LLMs) are advancing, with potential healthcare applications.
• LLM performance in general medical exams is studied, but rheumatology-specific evaluation is needed.

Purpose of the Study:

• To assess the performance of Chat Generative Pre-Trained Transformer (ChatGPT) on Israeli rheumatology board examination questions.
• Evaluate different ChatGPT model variants and prompting strategies.

Main Methods:

• 200 multiple-choice questions from 2023-2024 Israeli rheumatology board exams were used.
• Three GPT-4 Turbo variants were tested: base, few-shot chain-of-thought, and knowledge-augmented.
• Models were evaluated on English and Hebrew versions, with image-inclusive questions analyzed separately.

Main Results:

• The knowledge-augmented model achieved the highest accuracy (81%) on text-based questions.
• Performance on image-based questions was substantially lower across all models, ranging from 34.8% to 65.2%.
• No statistically significant differences were found between model performances overall.

Conclusions:

• LLMs demonstrate potential for rheumatology board exam assessment but have critical limitations.
• Future research must address challenges in image interpretation and complex case management for LLM application.
• Improving LLM capabilities in visual diagnosis is crucial for rheumatology.