IVCM-Insight: automated interactive interpretation of in vivo confocal microscopy

In vivo confocal microscopy (IVCM) is a critical ophthalmic examination that provides in vivo cytological and neurological information essential for diagnosing corneal and certain systemic diseases, but its clinical utility is limited by time-consuming interpretation and the need for subspecialty expertise. We developed IVCM-Insight, an artificial intelligence (AI) system integrating image-text contrastive learning with large language models (LLMs) for automated report generation and interactive question answering (QA). Based on 30,368 IVCM images and 4155 paired clinical reports, the model was trained with contrastive alignment, image-conditioned language modeling, and multi-image consistency loss to produce structured diagnostic reports while a domain-adapted LLM supported patient-centered QA. Automated evaluation showed strong agreement with the reference reports: Bilingual Evaluation Understudy (BLEU)-1 to BLEU-4 scores were 0.69, 0.58, 0.47, and 0.41, Recall-Oriented Understudy for Gisting Evaluation (ROUGE-L) was 0.67, Consensus-based Image Description Evaluation (CIDEr) was 1.85, and Metric for Evaluation of Translation with Explicit Ordering (METEOR) was 0.66. In addition, the multi-label classification achieved an accuracy of 0.96 and an F1 score of 0.80. Manual assessment by corneal specialists rated report accuracy (4.17), completeness (4.19), coherence (4.70), and diagnostic support (4.06), with excellent inter-rater reliability; QA outputs achieved high accuracy (4.33), relevance (4.54), and non-harmfulness (4.81). Representative cases, including cytomegalovirus, fungal, and Acanthamoeba keratitis, demonstrated accurate detection of key findings and clinically safe explanations. To our knowledge, IVCM-Insight is the first dedicated AI system for comprehensive IVCM interpretation, with potential to enhance diagnostic efficiency, strengthen physician-patient communication, and broaden access to advanced corneal imaging across care settings.