OphGLM: An ophthalmology large language-and-vision assistant.

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Computer Vision
• Natural Language Processing

Background:

• Computer-aided diagnostic methods in ophthalmology have limitations in human-computer interaction and clinical applicability due to restricted output formats.
• Existing solutions lack practical approaches for ophthalmic visual question answering before the advent of Large Language Models (LLMs).

Purpose of the Study:

• To address the limitations of current ophthalmic diagnostic tools by developing an advanced interactive system.
• To investigate and establish a novel Large Language Model (LLM)-based approach for ophthalmic visual question answering.

Main Methods:

• Construction of OphGLM, an ophthalmology large language-and-vision assistant, comprising an image encoder, text encoder, fusion module, and LLM module.
• Establishment of FundusTuning-CN, a new Chinese ophthalmic fine-tuning dataset including fundus instruction and conversation sets.
• Development of a novel LLM-tuning strategy using FundusTuning-CN to efficiently integrate visual understanding and ophthalmic knowledge into LLMs.

Main Results:

• OphGLM demonstrates superior visual understanding, outperforming open-source visual language models in fundus disease classification tasks.
• Leveraging FundusTuning-CN, OphGLM surpasses open-source medical LLMs in both ophthalmic knowledge and interactive capabilities.
• The proposed OphGLM shows significant potential for revolutionizing clinical applications in ophthalmology.

Conclusions:

• OphGLM represents a significant advancement in ophthalmic diagnostic interaction, bridging the gap between visual data and clinical decision-making.
• The developed LLM-tuning strategy and dataset offer a cost-effective and efficient method for enhancing medical LLMs with specialized knowledge.
• OphGLM has the potential to transform ophthalmic care through improved diagnostic accuracy and enhanced clinical applicability.