Performance of Publicly Available Large Language Models on Internal Medicine Board-style Questions

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Large Language Model (LLM) Evaluation

Background:

• Ongoing research benchmarks large language models (LLMs) against physician knowledge using medical examinations.
• Limited studies have assessed LLM performance on internal medicine (IM) board examination questions.
• The impact of domain-specific knowledge augmentation on LLM performance in medical contexts is not well-established.

Purpose of the Study:

• To evaluate the performance of various LLMs (GPT-3.5, GPT-4.0, LaMDA, Llama 2) on internal medicine board-style questions.
• To assess the effect of input augmentation using medical texts (Harrison's Principles of Internal Medicine) on LLM performance.
• To compare LLM performance against human respondents on IM board examination questions.

Main Methods:

• 240 randomly selected IM board-style questions from the Medical Knowledge Self-Assessment Program (MKSAP) were used.
• LLMs were accessed via API and chatbot interfaces, with and without Retrieval Augmented Generation (RAG) using Harrison's Principles of Internal Medicine.
• LLM-generated explanations for correct answers were compared to human-generated explanations by a blinded, board-certified physician.

Main Results:

• GPT-4.0 achieved the highest scores (77.5-80.7%), outperforming GPT-3.5, human respondents, LaMDA, and Llama 2.
• GPT-4.0 surpassed human MKSAP users across all IM subjects, with notable strengths in Infectious Disease and Rheumatology.
• Input augmentation via RAG increased GPT-3.5 and GPT-4.0 performance by 4.5-7.5% when accessed via API.

Conclusions:

• GPT-4.0 demonstrates superior performance on internal medicine board-style questions compared to other LLMs and human respondents.
• Retrieval Augmented Generation (RAG) is a viable technique for enhancing LLM accuracy in medical examinations by incorporating domain-specific knowledge.
• LLM performance can vary based on access method (API vs. chatbot), with chatbots generally showing higher scores.