Leakage-Aware Visit-Level Benchmarking Reveals Representational Overlap in Deep Learning for Bacterial Versus Fungal...

Area of Science:

• Ophthalmology
• Medical Imaging
• Artificial Intelligence

Background:

• Bacterial and fungal keratitis are serious eye infections.
• Accurate classification from slit-lamp images is crucial for timely treatment.
• Deep learning (DL) shows promise but requires robust benchmarking.

Purpose of the Study:

• To benchmark DL models for bacterial vs. fungal keratitis classification.
• To evaluate models using a leakage-aware, visit-level framework.
• To analyze embedding-space geometry for performance limitations.

Main Methods:

• Retrospective study of 101 patients (258 visits) with culture-confirmed keratitis.
• Compared image-level classifiers, multiple instance learning (MIL), and DINOv2 retrieval.
• Utilized full-frame, region of interest (ROI), and lesion-centered preprocessing.
• Analyzed embedding geometry with UMAP and assessed probability calibration.

Main Results:

• Visit-level AUROC ranged from 0.661 to 0.677.
• CNN-based MIL models achieved higher AUROC but showed greater overfitting.
• DINOv2 lesion-crop retrieval demonstrated stable generalization and superior calibration.
• Significant class overlap observed in embedding space (90.9% fungal visits within bacterial convex hull).

Conclusions:

• DL models achieved modest discrimination for keratitis classification.
• Retrieval-based models offered better generalization and calibration than MIL.
• Performance ceiling may be influenced by data representational overlap.
• Multicenter validation is needed to determine if findings are task-intrinsic or site-specific.