AI-assisted rheumatology triage changes with referral framing

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Rheumatology Workflow Optimization

Background:

• Physician adoption of healthcare AI is widespread, with large language models (LLMs) increasingly integrated into patient triage.
• LLMs influence rheumatology referral pathways, impacting patient access and speed of care.
• Understanding AI behavior in triage is critical for safe and equitable healthcare delivery.

Purpose of the Study:

• To evaluate the impact of specific cues in referral notes and patient descriptions on AI-assisted triage decisions.
• To determine if demographic factors or framing/descriptive elements cause greater shifts in AI triage outcomes.
• To assess the reliability of LLMs in rheumatology triage when clinical information is consistent but contextual details vary.

Main Methods:

• A controlled experiment using 30 rheumatology vignettes, each rephrased into 90 variants to test 9 dimensions.
• Evaluation of five LLMs from three major providers (Anthropic, Google, OpenAI) with over 200,000 queries.
• Analysis of 16 clinical fields against physician-validated ground truth, assessing concordance and shifts due to controlled modifications and personas.

Main Results:

• LLM concordance with expert ground truth was initially high (0.869) but sensitive to input variations.
• Framing and descriptive language, particularly patient anxiety, caused the largest distortions, tripling psychological attributions (13.1% vs 4.5%).
• Clinician anchoring in referral notes reduced concordance, especially for urgency; race, ethnicity, socioeconomic status, and language barriers had no detectable effect.

Conclusions:

• AI triage concordance is easily disrupted by the wording of referral notes and patient symptom descriptions, not patient demographics.
• AI systems must differentiate objective clinical data from subjective framing to prevent bias.
• Urgency and psychological attributions in AI triage should be auditable safety signals before widespread clinical adoption.