Using Large Language Models to Retrieve Critical Data from Clinical Processes and Business Rules.

Area of Science:

• Artificial Intelligence in Medicine
• Clinical Informatics
• Natural Language Processing

Background:

• Current clinical decision support systems often rely on complex, rule-based frameworks that require frequent updates and can be difficult to manage.
• These systems face challenges in adapting to evolving medical knowledge and diverse patient scenarios.

Purpose of the Study:

• To evaluate the efficacy of the large language model (LLM), LLaMA 2, in interpreting complex clinical process models (CPMs).
• To assess LLaMA 2's capability in providing accurate clinical recommendations based on these models.
• To determine the performance of LLMs in healthcare information retrieval using encoded clinical pathways.

Main Methods:

• LLaMA 2 was trained on clinical pathways encoded in DOT language and embedded using SentenceTransformer.
• The model processed hypothetical patient cases to generate diagnoses, further evaluation suggestions, and management steps.
• Token-level accuracy was measured by comparing LLM output against ground truth, assessing both node and edge accuracy.

Main Results:

• LLaMA 2 demonstrated high accuracy in retrieving diagnoses, suggesting evaluations, and outlining management steps based on CPMs.
• The average node accuracy achieved was 0.91 (SD ± 0.045).
• The average edge accuracy achieved was 0.92 (SD ± 0.122).

Conclusions:

• LLMs, specifically LLaMA 2, show significant potential for enhancing healthcare information retrieval from complex clinical models.
• The study validates the use of LLMs for interpreting clinical pathways and generating relevant recommendations.
• Future work should explore improving LLM interpretability and integration into clinical workflows.