LMOD: A Large Multimodal Ophthalmology Dataset and Benchmark for Large Vision-Language Models

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Medical Imaging

Background:

• Vision-threatening eye diseases pose a global health challenge, often diagnosed late.
• Large vision-language models (LVLMs) show promise in assisting with ophthalmic diagnosis and interpretation.
• Current benchmarks are insufficient for evaluating LVLMs in specialized ophthalmology applications.

Purpose of the Study:

• To introduce LMOD, a comprehensive benchmark for evaluating LVLMs in ophthalmology.
• To assess the performance of state-of-the-art LVLMs on ophthalmic tasks.
• To identify limitations and failure modes of LVLMs in ophthalmology.

Main Methods:

• Developed LMOD, a multimodal benchmark with 21,993 instances covering five imaging modalities and various data types.
• Benchmarked 13 LVLMs across different domains (closed-source, open-source, medical).
• Conducted systematic error analysis to identify LVLM failure modes.

Main Results:

• LVLMs exhibited a significant performance decrease in ophthalmology compared to other domains.
• Identified key failure modes including misclassification, hallucination, and lack of domain knowledge.
• Supervised neural networks achieved high accuracy on these tasks.

Conclusions:

• Existing LVLMs are not yet adequate for reliable ophthalmology applications.
• The development of ophthalmology-specific LVLMs requires robust benchmarks and further research.
• LMOD provides a critical resource for advancing AI in eye care.