Artificial Intelligence in Triaging Patient Questions: An Evaluation of a Large Language Model for Distal Radius...

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Decision Support Systems
• Natural Language Processing in Healthcare

Background:

• Large language models (LLMs) offer potential for clinical decision support but require rigorous validation.
• This study evaluated a knowledge and intelligence messaging interface (KIMI), an LLM enhanced with retrieval-augmented generation and specific clinical guidelines.
• KIMI was configured with American Academy of Orthopaedic Surgeons guidelines for distal radius fracture management.

Purpose of the Study:

• To assess the efficacy of KIMI in acuity triaging for distal radius fractures.
• To evaluate KIMI's ability to generate appropriate, patient-facing responses.
• To ensure the safety and reliability of LLM-generated clinical information.

Main Methods:

• 100 simulated patient queries for distal radius fractures were analyzed.
• Four clinical experts independently assessed KIMI responses for guideline concordance, safety, clarity, and acuity.
• Bayesian mixed-effects and ordered logistic regression models were employed for statistical analysis.

Main Results:

• KIMI responses achieved high ratings: 94.2% safety, 88.7% guideline concordance, and 93.7% clarity.
• Agreement between expert and LLM-assigned acuity was 62.9%, with surgical queries showing slightly higher safety and acuity agreement.
• LLM-assigned acuity was significantly associated with expert-assigned acuity (OR=2.66).

Conclusions:

• The knowledge and intelligence messaging interface (KIMI) demonstrated safe, clinically concordant, and clear communication.
• These findings support the feasibility of using enhanced LLMs for patient engagement in low-to-moderate risk care coordination.
• LLM-driven tools show promise for asynchronous patient communication in orthopedic care settings.