Related Experiment Video
Updated: Jun 12, 2025
03:14
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
516
Comparative Assessment of Otolaryngology Knowledge Among Large Language Models
Dante J Merlino1, Santiago R Brufau1, George Saieed1
1Department of Otolaryngology-Head and Neck Surgery, Mayo Clinic, Rochester, Minnesota, U.S.A.
The Laryngoscope
|September 21, 2024
Summary
OpenAI's GPT-4 demonstrated superior performance in answering otolaryngology questions compared to other large language models. Prompting for reasoning improved accuracy across all evaluated models, highlighting potential for AI in medical education.
Area of Science:
- Artificial Intelligence in Medicine
- Medical Education Technology
- Otolaryngology-Head and Neck Surgery
Background:
- Large language models (LLMs) are increasingly being explored for their potential in medical applications.
- Evaluating LLM performance on specialized medical knowledge is crucial for understanding their utility.
Purpose of the Study:
- To assess the accuracy of advanced large language models (OpenAI's GPT-3.5 and GPT-4, Google's PaLM2 and MedPaLM, Meta's Llama3:70b) on otolaryngology clinical test questions.
Main Methods:
- A dataset of 4566 otolaryngology multiple-choice questions was utilized.
- Each LLM received a standardized prompt followed by a question.
- 100 questions answered incorrectly by all models were analyzed for error patterns.
Main Results:
- GPT-4 achieved the highest accuracy (77.1%), followed by MedPaLM (70.6%), Llama3:70b (66.8%), GPT-3.5 (58.5%), and PaLM2 (56.5%).
- Providing reasoning prompts enhanced accuracy for all models, with GPT-4 showing the most significant improvement (31% of incorrect answers corrected).
Conclusions:
- LLMs exhibit varying capabilities in understanding otolaryngology clinical knowledge.
- GPT-4 demonstrates a strong grasp of otolaryngology concepts, making it a promising tool for head and neck surgery education with appropriate caution.
