Benchmarking Open-Source Vision Language Models in Orthopedic In-Training Examination: A Comparison with Residents,...

Advancing orthopedic care through large language models requires both multimodal processing capabilities for medical images and open-source deployment options for secure in-house operations, yet these remain underexplored in current literature. This study aims to benchmark open-source vision-language models (VLMs) against orthopedic residents using the Orthopedic In-Training Examination (OITE), assess domain-specific performance across orthopedic subspecialties, and investigate the relationship between model parameter size and performance.

Six open-source VLMs of varying sizes (Alibaba Qwen2.5-VL-72B-Instruct, Alibaba Qwen2.5-VL-32B-Instruct, Alibaba Qwen2.5-VL-7B-Instruct, Alibaba Qwen2.5-VL-3B-Instruct, Meta Llama-3.2-90B-Vision-Instruct, Meta Llama-3.2-11B-Vision-Instruct) were evaluated using the 2023 OITE (210 questions; 111 with images). Model performance was compared to resident scores from the 2023 OITE technical report. Pearson correlation coefficient was used to assess the association between model size and performance.

The 2 largest open-source models, Qwen2.5-VL-72B and Llama-3.2-90B, demonstrated performance levels comparable to those of second-year orthopedic residents on the OITE examination. A mid-sized model, Qwen-32B, slightly outscored first-year residents. In contrast, small-sized models (under 11 billion parameters) performed worse than first-year residents. Qwen2.5-VL-72B performed best in foot & ankle and sports medicine topics, while Llama-3.2-90B was strongest in basic science and hand & wrist. All models had the most difficulty with spine and pediatric questions. Overall, model accuracy increased steadily with model size up to 72 billion parameters, but larger sizes showed little additional improvement.

Smaller models offer reduced accuracy in exchange for lower hardware requirements. Spine and pediatric domains remain consistently areas of underperformance across all models. Model selection should be based on domain-specific benchmark results to balance clinical needs with hardware limitations. While promising, open-source VLMs currently require further refinement and validation before they can be reliably applied in clinical or educational settings.