Large Language Models in Biochemistry Education: Comparative Evaluation of Performance

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Large Language Models (LLMs)

Background:

• Advancements in AI and LLMs are revolutionizing medicine.
• LLMs show potential in passing medical board exams.
• Validation is needed for LLMs' ability to answer subject-specific medical questions.

Purpose of the Study:

• To compare the performance of advanced LLM chatbots against medical students in medical biochemistry.
• To analyze the accuracy of Claude, GPT-4, Gemini, and Copilot on biochemistry multiple-choice questions.

Main Methods:

• Utilized 200 United States Medical Licensing Examination (USMLE)-style multiple-choice questions (MCQs) covering 23 topics.
• Evaluated 5 attempts by Claude 3.5 Sonnet, GPT-4-1106, Gemini 1.5 Flash, and Copilot for accuracy.
• Employed chi-square tests for statistical analysis (P<.05).

Main Results:

• Chatbots averaged 81.1% accuracy, outperforming students by 8.3% (P=.02).
• Claude achieved the highest accuracy (92.5%), followed by GPT-4 (85%), Gemini (78.5%), and Copilot (64%).
• Top-performing topics included eicosanoids, bioenergetics, hexose monophosphate pathway, and ketone bodies.

Conclusions:

• AI models exhibit distinct strengths in specific medical domains, potentially aiding biochemistry education.
• LLM performance suggests significant potential for AI in medical education and assessment.
• AI tools can offer targeted support for medical students in specialized subjects.