Detecting emergencies in patient portal messages using large language models and knowledge graph-based...

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Clinical Decision Support Systems

Background:

• Patient messages via portals often require urgent attention, posing a patient safety risk if not promptly identified.
• Traditional methods for triaging patient messages can be time-consuming and may delay critical care.
• Large Language Models (LLMs) offer potential for automated message analysis but require domain-specific knowledge for accuracy.

Purpose of the Study:

• To develop and evaluate an approach using LLMs and knowledge graphs for triaging emergency patient messages.
• To improve patient safety by enabling timely notification for emergency cases.
• To guide patients towards immediate medical help when portal messages indicate an emergency.

Main Methods:

• Four models were developed: Prompt-Only LLM, Naïve Retrieval Augmented Generation (RAG), RAG from Knowledge Graph with Local Search, and RAG from Knowledge Graph with Global Search.
• A dataset of 1020 patient messages from Vanderbilt University Medical Center was used.
• A knowledge base derived from a nurse triage book with 225 protocols informed the RAG models.

Main Results:

• The RAG from Knowledge Graph model with global search demonstrated superior performance.
• This model achieved an accuracy of 0.99, sensitivity of 0.98, and specificity of 0.99.
• Significant improvements in triaging emergency messages were observed compared to LLM without RAG and naïve RAG.

Conclusions:

• LLMs augmented with domain-specific knowledge, particularly through structured knowledge graphs, enhance performance in triaging patient messages.
• The integration of LLMs with a nurse triage knowledge graph shows promise for effectively identifying emergency patient communications.
• Future work should involve expanding the knowledge graph and evaluating the system's real-world impact on patient outcomes.