ChatGPT Performs at the Level of a Third-Year Orthopaedic Surgery Resident on the Orthopaedic In-Training Examination

Area of Science:

• Artificial Intelligence in Medical Education
• Orthopaedic Surgery Training
• In-Training Examinations

Background:

• Publicly available AI language models show promise in text generation and problem-solving.
• The transformative potential of AI necessitates evaluation in specialized medical fields.
• Assessing AI performance on high-stakes medical examinations is crucial.

Purpose of the Study:

• To evaluate the performance of ChatGPT Plus on the 2021 Orthopaedic In-Training Examination (OITE).
• To compare ChatGPT Plus's OITE scores with those of national orthopaedic surgery residents.
• To explore the potential utility of AI in orthopaedic surgery education.

Main Methods:

• 201 questions from the 2021 OITE were administered to ChatGPT Plus (version 4.0).
• Media-containing questions relying solely on images were excluded.
• ChatGPT's performance was compared to the 2021 national resident norm table.

Main Results:

• ChatGPT Plus achieved an overall score of 61.2% (123/201).
• The AI model scored 64.2% on non-media questions.
• ChatGPT Plus's performance was equivalent to that of an average PGY3 orthopaedic surgery resident.

Conclusions:

• ChatGPT Plus can pass the OITE, demonstrating a proficiency level comparable to a third-year resident.
• The AI's ability to provide logical reasoning may enhance resident understanding of orthopaedic principles.
• Further research is needed to ascertain the long-term impact of AI on learning and board certification outcomes.