Evaluating large language models for specialist referral triage in primary care: a quantitative study using...

Effective and timely referral to specialist care is a fundamental responsibility of primary care providers, including family physicians, general practitioners, and community health workers. However, challenges in triage can lead to delays, unnecessary referrals, and increased strain on healthcare systems. Advances in artificial intelligence (AI) now offer new opportunities to support referral decision-making at the primary care level.

This study evaluates the performance and reliability of two AI models as referral decision support tools, using simulated scenarios commonly encountered in primary care settings involving ear, nose, and throat conditions.

Sixteen clinical vignettes representing common or high-stakes primary care presentations requiring specialist input were presented to each model in both structured clinical and informal patient-language formats. Responses were independently assessed by five otolaryngologists and 10 lay reviewers using a standardized rubric focused on appropriateness, clarity, safety, and usefulness. Quantitative analysis included comparisons of model performance, reviewer agreement, and the impact of prompt structure on output quality.

Both AI models generated safe and clinically appropriate referral recommendations when provided with structured clinical input. No statistically significant differences were observed between the two models across the evaluated domains. Performance declined for one model when prompts were presented in informal language, underscoring the importance of clear input structure. Reviewer agreement was high, confirming the reliability of findings.

AI decision support tools show potential to assist specialist referral triage in primary care. Clear, structured input is essential to maximize safety and reliability.