Secretary or SecretarAI: assessing the triage performance of AI vs human staff in a specialized dental clinic

Effective patient triage in specialized clinics is crucial for managing long waiting lists and mitigating clinical risk. This is often performed by non-clinical administrative staff, whose prioritization may differ from clinical experts. This study aimed to compare the triage performance of Large Language Models (LLMs) against human staff, using senior clinicians as the gold standard.

A cross-sectional survey study was conducted at a tertiary medical center. A custom survey presenting 19 clinical vignettes was administered to secretaries, dental assistants, and five distinct LLMs (Chat GPT-o3, Chat GPT-o4, Gemini Pro2.5, Gemini Flash2.5, OpenEvidence). Participants ranked hypothetical patients by urgency. An expert panel of senior clinicians established a gold-standard ranking for comparison. Agreement was assessed using Fleiss' κ and Cohen's weighted κ.

Artificial intelligence (AI) models demonstrated significantly higher agreement with the expert gold standard (Fleiss' κ = 0.572) compared to dental assistants (κ = 0.249) and secretaries (κ = 0.227). The AI cohort also showed superior internal consistency (Cronbach's α = 0.950). Among individual models, GPT-o3 (rationalization model) achieved the highest agreement (weighted κ = 0.681). Years of experience among human staff did not correlate with improved triage accuracy.

Contemporary LLMs can align more closely with expert clinical judgment in patient prioritization than non-clinical staff. While not a substitute for human oversight, AI shows significant promise as a reliable and consistent decision-support tool to augment existing triage processes, potentially enhancing patient safety and improving resource allocation in specialized care.