From Gram Stain to Decision Support: Performance of Multimodal Large Language Models in Blood Culture Microscopy

Multimodal large language models (LLMs) offer significant potential for image-based diagnostics, yet their reliability in routine clinical microbiology workflows, specifically Gram stain interpretation, remains insufficiently characterized. In this prospective diagnostic accuracy study, 100 Gram-stained blood culture smear images from routine laboratory workflows were evaluated by three multimodal LLMs (ChatGPT-4o, Gemini 2.5 Flash, and Claude Opus). Models were assessed using standardized zero-shot prompts across two independent runs to evaluate intramodel consistency. Performance was benchmarked against three expert medical microbiologists and automated organism identification (reference standard) across Gram type, major morphology, combined classification, and fine-grained sub-details. Expert microbiologists achieved 100% concordance with the reference standard for the predefined primary categories of Gram type and major morphology. LLM accuracy was high for Gram-type classification (95-98%) but lower for cellular morphology (84-85%), resulting in combined accuracies of 82-84%. No significant inter-model differences were observed in primary tasks. Intramodel consistency was high across runs (Cohen's κ = 0.73-0.96). However, performance declined for morphological sub-details, where ChatGPT and Gemini significantly outperformed Claude. Organism-level accuracy was high for Gram-positive cocci and Gram-negative bacilli but substantially lower for Gram-positive bacilli and yeast. Multimodal large language models show promising baseline performance in Gram-stained blood culture interpretation, particularly for Gram type and major morphology. However, reduced accuracy in combined interpretation, fine-grained morphology, and less common organism classes highlights current limitations of out-of-the-box models. These findings support the use of multimodal LLMs as expert-supervised decision-support tools rather than standalone systems and emphasize the need for task-specific optimization and multicenter validation before routine clinical implementation.