Comparison of the Performance of Five Generative Artificial Intelligence Models on a Medical Molecular Biology...

Objective The aim of this study is to evaluate the performance of five common Chinese generative artificial intelligence (GAI) models on a medical molecular biology examination and assess the application value of these GAIs in teaching. Methods A set of medical molecular biology test questions was used to measure the performance of five Chinese GAIs, including ERNIE Bot, chatGLM, iFLYTEK Spark, Qwen, and Doubao. The correct response rates of the five GAIs were compared with those of actual medical undergraduates using an unpaired t-test in GraphPad Prism 6.01. Results The total scores of the five GAIs all exceeded the passing score of 60 (full score: 100), ranging from 75.67 to 88.67. ERNIE Bot, chatGLM, Qwen, and Doubao demonstrated higher correct rates for total scores (80.33%-88.67%; p-value: 0.0127-0.0492) and for multiple-choice questions (83.33%-87.50%; p-value: 0.0071-0.0137) compared to actual undergraduates, showing a different distribution pattern of incorrect responses. Conclusion This study demonstrated the effectiveness of the five GAIs as learning aids in medical molecular biology. However, due to occasional incorrect answers, undergraduates should apply critical thinking when using GAI-generated responses. Meanwhile, a discipline-specific AI agent for medical molecular biology should be developed as soon as possible.