Related Experiment Video
Updated: Jun 3, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Examining the Performance of ChatGPT in Comprehensive Pre-Internship Exam: The Potential of Artificial Intelligence
Michaeel Motaghi Niko1, Zahra Karbasi2, Maryam Kazemi3
1Department of Nursing Shahrekord University of Medical Sciences Shahrekord Iran.
Background And Aims:
ChatGPT is a popular large language model with potential educational applications in medicine. However, its performance in standardized, multi-disciplinary medical exams has not been comprehensively assessed. This study evaluates ChatGPT's accuracy and quality in Iran's national medical pre-internship exam.
Methods:
We tested ChatGPT (GPT-3.5, May 3rd version) on 195 multiple-choice questions from the March 2022 Iranian pre-internship exam, covering 23 medical specialties. Questions with visual content were excluded. Each question was asked in a new chat to avoid memory bias. Responses were evaluated by 55 experts using a 5-point Likert scale and compared against the official answer key. Data were analyzed descriptively using SPSS.
Results:
ChatGPT answered 68.6% of questions correctly. Expert ratings averaged 4.23/5 (SD = 1.21), indicating good to excellent quality. Best-performing specialties included pharmacology (85.7%), otorhinolaryngology (83.3%), and dermatology (83.3%). Lower performance was observed in pulmonology (42.9%) and epidemiology (50%).
Conclusion:
ChatGPT shows promise as a supplemental educational tool in medical education, but its accuracy varies by specialty. Faculty guidance is essential to ensure responsible integration until further improvements and validations are made.
