Evaluating the Accuracy of Large Language Models in Answering Asthma Multiple Choice and Objective Structured...

Large language models (LLMs) are demonstrating increasing promise across clinical applications, but their domain-specific knowledge in asthma has not been thoroughly explored. Additionally, state-of-the-art models released in 2025 (ChatGPT-5, ChatGPT-o3, Claude-3.7, DeepSeek-V3, and Grok-3) have yet to be studied in asthma.

How accurate are LLMs at answering adult asthma multiple choice questions (MCQs) and Objective Structured Clinical Examinations (OSCEs)?

Fourteen LLM chatbots (Asthma-GPT, ChatGPT-4o, ChatGPT-o1, ChatGPT-o3-mini, ChatGPT-o3-mini-high, ChatGPT-5, Claude-3.7-Sonnet, DeepSeek-V3, Gemini-2.0-Flash, Grok-3, Le-Chat, Llama-3, NICE-Asthma-Bot, and Biomistral) were tested in 5 iterations on 116 adult asthma MCQs and 3 asthma OSCEs. Using generalized linear mixed-effects models, we compared accuracy between the LLMs using log-odds scale and pairwise comparisons with Tukey adjustment, and assessed difference in MCQs vs OSCEs, generic vs medicine-specific models, open-source vs proprietary models, and patient- vs clinician-oriented questions.

MCQ accuracy was excellent, with 13 of 14 models exceeding 85%, and ChatGPT-o1, Asthma-GPT, NICE-Asthma-Bot, ChatGPT-5, and ChatGPT-4o surpassing 95% accuracy. OSCE accuracy varied more significantly, but most models still achieved 70% to 86% accuracy. ChatGPT-o1 ranked highest in MCQs, and ChatGPT-5 ranked highest in OSCEs. Generalized linear mixed-effects model revealed that LLMs performed better at MCQs than OSCEs (92.8% vs 71.1%, P < .001), and that LLMs performed better at patient-oriented MCQs than clinician-oriented MCQs (96.3% vs 85.6%, P < .01). No significant differences were found between open-source and proprietary models in MCQs, or between generic and medicine-specific LLMs.

Our results indicate that current LLMs demonstrate high accuracy across asthma-related clinical and patient-oriented questions, particularly the ChatGPT family of models. These models may be useful tools to integrate into future patient and clinician education interventions.