Evaluating the diagnostic performance of multimodal large language models in thyroid cytology

Thyroid fine-needle aspiration (FNA) cytology is a widely used, cost-effective method for evaluating thyroid nodules. However, interpretation of thyroid FNA smears remains challenging, particularly in settings with limited access to experienced cytopathologists. Multimodal large language models (LLMs) have shown potential in static image interpretation in histopathology and may serve as supportive tools when expert review is unavailable. Nevertheless, data regarding their application in cytopathology, including thyroid cytology, remain limited. In this study, we evaluated the diagnostic accuracy and consistency of ChatGPT and Gemini in interpreting 100 thyroid FNA cytology images, comprising 50 cases of papillary thyroid carcinoma (PTC) and 50 non-PTC lesions, including follicular nodular disease, follicular neoplasm, medullary thyroid carcinoma, and anaplastic thyroid carcinoma. Representative images were captured at 40× magnification and reviewed by expert pathologists to confirm the reference diagnoses. Each image was analyzed by ChatGPT and Gemini across five separate rounds. For the class-based analysis, ChatGPT and Gemini achieved mean diagnostic accuracies of 26.2% and 23.6%, respectively, based on Bethesda categories, and 39.2% and 20.2%, respectively, for specific diagnostic entities. ChatGPT rendered a higher number of suspicious for malignancy and atypia of undetermined significance diagnoses than Gemini. Five-image submission and inclusion of TIRADS data improved performance metrics in some instances. Medullary and anaplastic thyroid carcinomas were rarely correctly identified. In conclusion, although ChatGPT and Gemini demonstrated the potential of LLMs in cytologic image interpretation and drafting descriptive reports, their diagnostic performance in thyroid FNA cytology was limited, underscoring the current constraints of LLMs in this domain.