ChatGPT's Performance on Portuguese Medical Examination Questions: Comparative Analysis of ChatGPT-3.5 Turbo and...

Area of Science:

• Artificial Intelligence in Medical Education
• Natural Language Processing Applications
• Healthcare Professional Assessment Tools

Background:

• ChatGPT advancements offer new avenues for medical education assessment and learning.
• AI tools like ChatGPT can potentially enhance doctor evaluations and instructional effectiveness.

Purpose of the Study:

• To evaluate the performance and consistency of ChatGPT-3.5 Turbo and ChatGPT-4o mini on European Portuguese medical examination questions.
• To compare AI model performance against human candidates in a standardized medical assessment.

Main Methods:

• ChatGPT-3.5 Turbo and ChatGPT-4o mini were tested on the 2023 National Examination for Access to Specialized Training (PNA).
• Consistency was assessed by re-prompting models with "Are you sure?"; response differences were analyzed.
• Statistical analysis included McNemar test, t tests, and descriptive statistics (frequencies, percentages, means, CIs).

Main Results:

• ChatGPT-4o mini achieved 65% accuracy on the PNA exam, outperforming ChatGPT-3.5 Turbo.
• ChatGPT-4o mini surpassed the performance of human medical candidates.
• ChatGPT-3.5 Turbo showed moderate performance in the assessment.

Conclusions:

• ChatGPT models exhibit significant advancements and potential utility in medical education.
• Careful implementation, teacher oversight, and further research are crucial for integrating AI in medical training.