Rise of the Machines: Comparing Performance of Artificial Intelligence Large Language Models on Pharmacy Specialty...

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Decision Support Systems
• Pharmacy Practice

Background:

• Large language models (LLMs) are increasingly utilized for clinical information retrieval and decision support.
• Comparative performance of LLMs on specialized pharmacy content is not well-understood.

Purpose of the Study:

• To evaluate the performance of 15 large language models (LLMs) on Board of Pharmacy Specialties (BPS) certification practice questions.
• To identify differences in LLM accuracy across 14 pharmacy specialty domains.

Main Methods:

• 145 publicly available BPS certification practice questions across 14 specialties were used.
• 15 LLMs were tested using a standardized prompt without prompt engineering.
• Model responses were scored against official answer keys, with statistical analysis performed on accuracy differences.

Main Results:

• Mean accuracy across all LLMs was 86.2%, with individual model accuracy ranging from 79.3% to 91.7%.
• Significant differences in performance were observed among LLMs (p < 0.001).
• Microsoft Copilot (GPT-5), Google Gemini 2.5 Flash, and OpenAI o3 (Reasoning) showed superior performance compared to Perplexity AI.

Conclusions:

• LLMs demonstrate high accuracy on Board of Pharmacy Specialties certification practice questions.
• Variability in performance across LLMs and specialty domains was limited.
• Findings support further investigation of LLMs for pharmacy practice and clinical decision support, highlighting the need for domain-specific validation.