ChatGPT-3.5 and the Polish thoracic surgery specialty examination: a performance evaluation

Area of Science:

• Medical Education
• Artificial Intelligence in Medicine
• Thoracic Surgery

Background:

• Rapid advancements in artificial intelligence (AI) present new opportunities for medical applications.
• The reliability and limitations of AI in healthcare require thorough evaluation.
• Assessing AI's performance in specialized medical examinations is crucial.

Purpose of the Study:

• To evaluate the effectiveness of the ChatGPT-3.5 language model.
• To assess ChatGPT-3.5's performance on the National Specialist Examination (PES) for thoracic surgery.

Main Methods:

• Analysis of 120 test questions from the 2015 PES examination.
• Questions categorized by subject matter, clinical character, and cognitive requirements.
• ChatGPT-3.5 answered each question five times; statistical tests (χ², Kruskal-Wallis, Mann-Whitney, Spearman) and Fleiss' k coefficient were used.

Main Results:

• ChatGPT-3.5 achieved 42.2% correct answers, falling short of the 60% passing score.
• A statistically significant difference was noted between clinical and non-clinical questions (p=0.041).
• Correct answers had higher confidence (p<0.001), but confidence did not correlate with psychometric indicators; response consistency was moderate (k=0.341).

Conclusions:

• ChatGPT-3.5's performance is equivalent to a failing score on the specialist examination.
• While response confidence correlates with correctness, AI limitations in clinical knowledge and consistency necessitate caution.
• Further research is needed to understand AI's role in specialized medical knowledge assessment.