The model student: GPT-4 performance on graduate biomedical science exams

Daniel Stribling ^1,2,3, Yuxing Xia ^4,5, Maha K Amer ⁶

¹Department of Molecular Genetics and Microbiology, University of Florida, Gainesville, FL, 32610, USA. ds@ufl.edu.
²UF Genetics Institute, University of Florida, Gainesville, FL, 32610, USA. ds@ufl.edu.
³UF Health Cancer Center, University of Florida, Gainesville, FL, 32610, USA. ds@ufl.edu.
⁴Department of Neuroscience, Center for Translational Research in Neurodegenerative Disease, College of Medicine, University of Florida, Gainesville, FL, 32610, USA.
⁵Department of Neurology, UCLA, Los Angeles, CA, 90095, USA.
⁶Department of Molecular Genetics and Microbiology, University of Florida, Gainesville, FL, 32610, USA.
⁷Department of Computer and Information Science and Engineering, Herbert Wertheim College of Engineering, University of Florida, Gainesville, FL, 32610, USA.
⁸UF Genetics Institute, University of Florida, Gainesville, FL, 32610, USA.
⁹Department of Anthropology, University of Florida, Gainesville, FL, 32610, USA.
¹⁰Department of Molecular Genetics and Microbiology, University of Florida, Gainesville, FL, 32610, USA. rrenne@ufl.edu.
¹¹UF Genetics Institute, University of Florida, Gainesville, FL, 32610, USA. rrenne@ufl.edu.
¹²UF Health Cancer Center, University of Florida, Gainesville, FL, 32610, USA. rrenne@ufl.edu.

⁰Department of Molecular Genetics and Microbiology, University of Florida, Gainesville, FL, 32610, USA. ds@ufl.edu.

Scientific Reports +

|

March 7, 2024

View abstract on PubMed

Summary

Large language models like GPT-4 show strong performance on biomedical science exams, outscoring students on many. However, limitations in handling figures and potential plagiarism require careful consideration for academic use.

Area Of Science

Biomedical Sciences
Artificial Intelligence
Educational Technology

Background

Large language models (LLMs) like GPT-4 and ChatGPT are increasingly capable text generators.
GPT-4 has demonstrated proficiency in standardized tests, but its trustworthiness in diverse knowledge domains needs evaluation.
Assessing AI performance in specialized fields like biomedical sciences is crucial for understanding its potential and limitations.

Purpose Of The Study

To evaluate the performance and accuracy of the GPT-4 large language model on graduate-level biomedical science examinations.
To identify specific question formats and content types where GPT-4 excels or struggles.
To inform the future design of academic assessments in the context of advanced AI tools.

Main Methods

GPT-4 was tested on nine graduate-level biomedical science exams, including seven that were blinded.
Performance was analyzed across different question types: fill-in-the-blank, short-answer, essay, and questions involving figures.
Responses were assessed for accuracy, plagiarism, and instances of hallucination.

Main Results

GPT-4 surpassed the student average score in seven out of nine exams and exceeded all student scores in four exams.
The model performed well on text-based questions and questions with figures from published manuscripts.
Poor performance was observed on questions with figures containing simulated data and those requiring hand-drawn answers. Plagiarism and hallucinations were noted in some responses.

Conclusions

GPT-4 demonstrates significant capabilities in answering graduate-level biomedical science questions, often outperforming human students.
The model's limitations, particularly with visual data and potential for generating inaccurate or plagiarized content, highlight the need for careful integration into academic settings.
Future academic assessments may need adaptation to account for AI capabilities and mitigate potential misuse.

Related Concept Videos

Serum Laboratory Studies, Stool Test, Breath Test

Gastrointestinal (GI) diagnostic studies are pivotal in confirming, ruling out, diagnosing, or staging various diseases, including cancers. Following diagnosis, allocating time for discussions with the patient and providing informational resources is crucial. Diagnostic assessments of the GI tract often occur in outpatient settings like endoscopy suites or GI labs. Preparation for these tests may include dietary restrictions, fasting, liquid bowel preparations, laxatives, enemas, and the...