Table-based language models for ophthalmology assessment in the emergency department

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Health Informatics

Background:

• General-domain large language models (LLMs) are increasingly used in healthcare.
• Their application in ophthalmology, particularly with tabular data from electronic medical records (EMRs), remains underexplored.

Purpose of the Study:

• To evaluate the diagnostic and assessment performance of OpenAI's Generative Pre-trained Transformer 4o (GPT-4o) on tabular ophthalmology data.
• To assess GPT-4o's ability to interpret real-world emergency department (ED) eye-related encounters.

Main Methods:

• GPT-4o processed 1,419 eye-related ED encounters from EMRs in tabular format using chain-of-thought (CoT) prompting.
• Performance was evaluated by board-certified ophthalmologists, assessing diagnosis and assessment accuracy.
• Inter-grader agreement was assessed, and GPT-4o's ability to identify documentation inconsistencies was tested.

Main Results:

• GPT-4o achieved an overall accuracy of 0.76 (95% CI, 0.74-0.79) in diagnosing and assessing ophthalmology encounters.
• Accuracy remained consistent regardless of the amount of input data (chief complaint, history, examination).
• After excluding encounters with unsupported documentation or requiring ancillary tests, GPT-4o's accuracy improved to 0.87 (95% CI, 0.85-0.89).

Conclusions:

• GPT-4o demonstrated proficiency in synthesizing tabular ophthalmology data for accurate diagnoses and assessments.
• The model successfully identified encounters lacking supporting EMR documentation.
• This LLM capability holds potential to aid clinicians in diagnosis within ophthalmology.