Illusion of competence: vision-language models provide confident but inaccurate explanations in cytological...

Large vision-language models (LVLMs) have shown impressive image-understanding capabilities across domains. However, their suitability for cytomorphological diagnostics remains unclear. Here, we systematically evaluated four state-of-the-art generalist LVLMs, GPT-4o, Gemini-2.0, Llama-3.2, and DeepSeek-VL2, and three biomedical LVLMs, LLaVA-Med, CONCH, and BiomedCLIP, across key cytomorphology benchmarks, including peripheral blood cell classification, morphology assessment, bone marrow cell classification, and cervical smear malignancy detection. Performance was assessed under zero-shot, few-shot, and fine-tuned settings. In zero-shot and few-shot evaluations, LVLMs performed poorly, often approaching random performance. In peripheral blood cell classification, GPT-4o achieved a zero-shot F1 score of only 0.22 ± 0.02 and a few-shot F1 score of 0.36 ± 0.03. Even after fine-tuning, GPT-4o was outperformed by a lightweight, dedicated hematology model. Beyond classification accuracy, we assessed interpretability and trustworthiness. Although LVLMs generated textual justifications, these often reflected textbook knowledge rather than the actual morphological features present in the cell images. Expert evaluation showed that 30% of explanations for misclassified cells were rated as poor or misleading. While LVLMs could segment cellular structures such as nuclei and granules, they failed to reliably identify the image regions relevant to their classification decisions. Our findings underscore three major limitations of current LVLMs in cytomorphology: (1) low diagnostic accuracy, (2) poor generalizability across domains, and (3) unreliable explainability. These results suggest that LVLMs require substantial improvement before they can be used for cell-type classification and morphology characterization in diagnostic settings. Purpose-built models remain the more effective and trustworthy choice.