Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Biomedical Data Science

Background:

• Electronic health records (EHRs) are vital but complex, risking diagnostic inaccuracies.
• Large language models (LLMs) show promise but require careful application in healthcare.
• Integrating knowledge graphs (KGs) with LLMs can enhance diagnostic reasoning with structured medical information.

Purpose of the Study:

• Introduce DR.KNOWS (Diagnostic Reasoning Knowledge Graph System).
• Integrate Unified Medical Language System-based KGs with LLMs.
• Improve diagnostic predictions from EHR data via contextually relevant path retrieval.

Main Methods:

• Utilized a stack graph isomorphism network for node embedding.
• Employed an attention-based path ranker to identify relevant knowledge paths.
• Evaluated DR.KNOWS on real-world EHR datasets against baseline LLMs and QuickUMLS.
• Implemented a human evaluation framework focused on clinical safety metrics.

Main Results:

• DR.KNOWS improved diagnostic concept extraction and prediction metrics.
• KG-integrated LLMs achieved superior ROUGE-L and concept F1-scores.
• Human evaluations confirmed DR.KNOWS' diagnostic rationales aligned with clinical reasoning.
• Addressed KG data biases through case-specific path selection.

Conclusions:

• DR.KNOWS enhances diagnostic accuracy and reasoning in clinical workflows.
• Represents progress toward trustworthy AI-driven clinical decision support.
• Further research needed for KG bias mitigation and generalizability.