Performance of an Intelligent Messaging Tool for Clinical Communications

Delays in patient care and occupational burnout are potential consequences of difficulty in managing rising volumes of patient portal messages. The Smart Messaging Tool (SMT) was a custom-built natural language processing tool developed to address 4 key criteria: alignment with care pathways, adaptation to evolving conditions, demonstration of high accuracy, and accommodation of large volumes of messages.

To assess the viability of SMT in classifying patient messages in a large, integrated, value-based health care system.

This quality improvement study compared the first 2 years of SMT implementation (March 28, 2023, to March 27, 2025) with the legacy system it replaced. Patient messages sent via the Southern California Permanente Medical Group's (SCPMG's) patient portal were processed by SMT, which generated a label that populated the topic column in the receiving clinician's inbox. Deployed broadly to primary care departments across SCPMG, SMT reached the inboxes of more than 1000 clinicians. Messages could come from any of SCPMG's 4.9 million patients who opted to send secure messages.

Median time to first read by a clinician of high-acuity messages and SMT classification performance metrics were assessed.

The SMT processed 3 030 247 messages sent by 1 042 418 unique patients (the majority [31.9%] aged 30-49 years; 60.5% female). Median time-to-first read by a clinician for high-acuity messages fell from 22.03 hours (95% CI, 21.47-22.48 hours) to 5.02 hours (95% CI, 4.50-5.77 hours). The SMT achieved 81.0% (95% CI, 77.8%-83.6%) accuracy, which is higher than the 44.0% accuracy of the legacy system. The SMT achieved a top-3 accuracy of 88.5% (95% CI, 86.0%-90.8%) and a top-5 accuracy of 90.7% (95% CI, 88.5%-92.7%).

This quality improvement study found that SMT reliably categorized patient messages to support improvements in the timely processing of high-acuity messages. The findings suggest that the SMT's practical applicability underscores its relevance to organizations aiming to leverage natural language processing to address message management challenges.