EYE-Llama, an in-domain large language model for ophthalmology

Area of Science:

• Artificial Intelligence in Medicine
• Ophthalmology Natural Language Processing
• Medical Question Answering Systems

Background:

• Large Language Models (LLMs) require in-domain data for enhanced performance in specialized fields.
• Accurate question-answering (QA) systems are crucial for clinical decision-making and patient education.

Purpose of the Study:

• To develop and evaluate LLMs specifically trained on ophthalmic datasets for improved medical QA.
• To introduce an open-source ophthalmic language dataset for model training.

Main Methods:

• Pre-trained LLMs (EYE-Llama) on a curated ophthalmology corpus (abstracts, textbooks, EyeWiki, Wikipedia).
• Fine-tuned models using diverse QA datasets.
• Compared EYE-Llama against Llama 2, ChatDoctor, and ChatGPT (GPT3.5) using four test sets.
• Evaluated models quantitatively (Accuracy, F1 score, BERTScore) and qualitatively by ophthalmologists.

Main Results:

• EYE-Llama achieved comparable F1 scores to ChatGPT on the AAO test set, outperforming Llama 2 and ChatDoctor.
• Fine-tuned models showed higher accuracy than Llama 2 and ChatDoctor on the MedMCQA dataset.
• EYE-Llama demonstrated superior accuracy on the PubmedQA dataset compared to Llama 2, ChatGPT, and ChatDoctor.

Conclusions:

• Pre-training and fine-tuning LLMs on domain-specific data significantly enhances medical QA capabilities.
• EYE-Llama models demonstrate the effectiveness of specialized LLMs in ophthalmology.
• The developed ophthalmic dataset and models offer valuable resources for advancing AI in eye care.