Related Experiment Video
Updated: Oct 19, 2026
Augmenting Large Language Models via Vector Embeddings to Improve Domain-Specific Responsiveness
Published on: December 6, 2024
Performance Evaluation of the Latest General-Purpose Large Language Models in Medical Reasoning and Clinical
Masaki Sakamoto1, Shintaro Honda1, Bungo Otsuki1
1Department of Orthopaedic Surgery, Graduate School of Medicine, Kyoto University, Kyoto, Japan.
Study Design:
Validation Study.
Introduction:
General-purpose large language models (LLMs) have shown substantial improvements since 2023, but their performance in specialized clinical tasks remains inadequately validated. This study aimed to evaluate the performance of the latest general-purpose LLMs as of December 2025 in both knowledge-based and practical tasks, specifically focusing on their utility in literature screening for orthopedic research requiring specialized judgment.
Methods:
We evaluated nine of the latest general-purpose LLMs from three major providers (OpenAI: gpt-5/5-mini/5-nano, Google: gemini-2.5-pro/flash/flash-lite, Anthropic: claude-opus-4-5/sonnet-4-5/haiku-4-5) using two approaches: (1) knowledge-based assessment using the MedQA-USMLE (United States Medical Licensing Examination) benchmark (1,273 questions), and (2) practical assessment by replicating the primary screening task from a published orthopedic meta-analysis (384 articles). All evaluations were conducted through an identical automated pipeline using structured output to ensure fair comparison.
Results:
Top-tier models achieved accuracy exceeding 0.94 (94%) on MedQA-USMLE (gpt-5: 0.9427 [94.27%], gemini-2.5-pro: 0.9340 [93.40%], claude-opus-4-5: 0.9026 [90.26%]), surpassing prior benchmarks. In the specialized meta-analysis screening task, the highest-tier models achieved performance approaching the agreement between two expert evaluators used as the reference standard (claude-opus-4-5: κ=0.8881, gpt-5: κ=0.8805 vs. inter-evaluator κ=0.8951). Confidence scores from structured output demonstrated trends suggesting utility for implementing quality-controlled two-stage screening.
Conclusions:
General-purpose LLMs have achieved high medical reasoning capabilities without medical-specific training. In orthopedic literature screening, the highest-tier models demonstrated performance approaching inter-expert agreement, with confidence metrics showing utility for the implementation of quality-controlled screening workflows. These findings suggest that LLMs may serve as potentially useful tools to reduce expert burden in medical literature screening, although validation was limited to a single orthopedic meta-analysis, and generalizability to other settings requires further investigation.