Input structure-driven instability and convergence in large language model clinical reasoning: a formative study...

Area of Science:

• Artificial Intelligence in Medical Education
• Clinical Reasoning Assessment
• Large Language Model (LLM) Performance

Background:

• Large language models (LLMs) are increasingly used in medical education and clinical settings.
• Prior research focused on LLM accuracy on exams, but not on the stability of clinical reasoning with varied input structures.
• Input structure is critical for safe educational and clinical deployment of LLMs.

Purpose of the Study:

• To investigate how question delivery structure affects performance stability, inter-model variability, and reproducibility of LLM clinical reasoning.
• To evaluate contemporary LLMs using pediatric residency-level multiple-choice questions (MCQs).

Main Methods:

• Generated 77 validated pediatric USMLE Step 2/3-style MCQs emphasizing diagnostic, management, and ethical reasoning.
• Evaluated six publicly available LLMs (October-December 2025 versions) under two conditions: simultaneous presentation vs. sequential delivery in batches of ten.
• Compared accuracy and inter-model variability using paired t-tests and one-way ANOVA.

Main Results:

• Simultaneous question presentation showed wide accuracy variation (38%-90%) and poor reproducibility across models.
• Sequential batch delivery improved performance convergence (83%-88%) with no significant inter-model differences.
• Batch delivery substantially reduced performance dispersion and instability across all evaluated LLMs.

Conclusions:

• LLM clinical reasoning is highly sensitive to input structure; prompt structure is key to reliable behavior.
• Structured batch delivery minimizes contextual load, improving reproducibility and reducing inter-model variability.
• Consider prompt structure in designing AI-supported medical education and assessment systems for reliable LLM use.