Comparative Performance of Large Language Models in Ophthalmology Referral Triage.

Area of Science:

• Ophthalmology
• Artificial Intelligence
• Natural Language Processing

Background:

• Automated classification of medical referrals can improve efficiency.
• Large Language Models (LLMs) offer potential for analyzing clinical text.

Purpose of the Study:

• To evaluate the classification accuracy and consistency of five advanced LLMs in categorizing Portuguese ophthalmology referral vignettes.
• To assess the impact of supervised in-context learning on LLM performance for these referrals.

Main Methods:

• 3,831 Portuguese ophthalmology referral vignettes were classified by five LLMs (ChatGPT 4o, ChatGPT 5.1, Perplexity Pro, Claude Sonnet 4.5, Claude Opus 4.1).
• A zero-shot prompting strategy was used initially, followed by a phase with in-context learning (957 labeled examples).
• Classification accuracy, consistency, and Fleiss' kappa agreement were calculated.

Main Results:

• Baseline accuracy averaged 68.7%, improving to 73.4% after in-context learning.
• ChatGPT 5.1 achieved peak accuracy (79.5%); ChatGPT 4o showed the largest consistency gain (66.8% to 93.8%).
• Performance exceeded 90% for common categories (e.g., diabetic screening) but was lower for rare/ambiguous complaints. Inter-run agreement ranged from moderate to substantial (κ = 0.462-0.801).

Conclusions:

• Advanced LLMs demonstrate significant potential in interpreting ophthalmology referrals, with notable accuracy and consistency improvements via in-context learning.
• LLM performance is lower for rare or ambiguous referral types, highlighting the need for careful implementation.
• LLMs could serve as scalable, low-cost triage aids in ophthalmology, contingent on human oversight and further clinical validation.