Harnessing advanced large language models in otolaryngology board examinations: an investigation using python and...

Area of Science:

• Artificial Intelligence in Medicine
• Medical Education Technology
• Otolaryngology Diagnostics

Background:

• Large language models (LLMs) are increasingly evaluated for specialized medical applications.
• Assessing LLM performance on board certification exams is crucial for understanding their utility.
• Previous evaluations of GPT-3.5 Turbo provide a baseline for longitudinal performance tracking.

Purpose of the Study:

• To evaluate the capabilities of advanced LLMs (GPT-4 variants, Gemini, Claude) on otolaryngology board exam questions.
• To assess the longitudinal performance changes of GPT-3.5 Turbo over one year.
• To compare the accuracy of different LLM series in a specialized medical domain.

Main Methods:

• A question bank of 2,576 otolaryngology board certification questions was used.
• Eleven LLMs, including GPT-3.5 Turbo, GPT-4 variants, Gemini, and Claude, were tested.
• Questions were administered via APIs using Python scripts for data collection.

Main Results:

• GPT-4o achieved the highest accuracy, particularly in allergology and head and neck tumors.
• Claude models were competitive but generally underperformed GPT-4 variants.
• GPT-3.5 Turbo showed a significant decline in accuracy compared to the previous year.
• Single-choice questions yielded higher accuracy than multiple-choice questions across all models.

Conclusions:

• Newer LLMs demonstrate significant potential for specialized medical content and certification.
• The performance decline in GPT-3.5 Turbo highlights the need for continuous LLM evaluation.
• Ongoing optimization and efficient API use are essential for advancing LLMs in medical education.