Clinical Reasoning and Knowledge Assessment of Rheumatology Residents Compared to AI Models: A Pilot Study

Area of Science:

• Medical Artificial Intelligence
• Clinical Decision Support Systems
• Rheumatology Research

Background:

• Artificial intelligence (AI) is increasingly integrated into medicine, evolving from rule-based systems to advanced models with potential in clinical decision-making.
• This study assesses the psychological impact of AI collaboration, specifically its role as a support tool for medical residents.
• AI's potential in clinical reasoning and management across various stages of patient care is explored.

Purpose of the Study:

• To compare the clinical decision-making abilities of junior rheumatology residents against trained and untrained AI models (GPT-4).
• To evaluate performance in clinical reasoning, pre-diagnosis, and first- and second-line management of inflammatory arthritis cases.
• To assess residents' attitudes toward AI integration and its perceived benefits in clinical practice.

Main Methods:

• Ten junior rheumatology residents and two GPT-4 models (trained and untrained) responded to 10 complex inflammatory arthritis cases.
• Performance was evaluated using the Revised-IDEA (R-IDEA) scoring system and additional case management metrics.
• Residents' attitudes toward AI were assessed via a structured questionnaire post-review of AI performance.

Main Results:

• Trained GPT-4 demonstrated superior performance across all case stages, achieving higher R-IDEA scores than residents.
• The AI model significantly outperformed residents in pre-diagnosis, first-line, and second-line management phases.
• A majority of residents (60%) expressed positive attitudes, viewing AI as a valuable support tool for competence and reducing fatigue/burnout.

Conclusions:

• Trained AI models, specifically GPT-4, exhibit higher proficiency than junior residents in rheumatology clinical reasoning and case management.
• Residents' positive reception suggests AI's potential as a supplementary tool to boost confidence and mitigate uncertainty in early residency.
• AI integration may serve as a valuable resource for medical residents during their initial training years.