From Conversation to Standardized Terminology: An LLM-RAG Approach for Automated Health Problem Identification in...

Area of Science:

• Natural Language Processing in Healthcare
• Clinical Informatics
• Artificial Intelligence in Medicine

Background:

• Ambient listening systems are increasingly used in healthcare for analyzing clinician-patient conversations.
• The Omaha System is a standardized terminology for patient care documentation, but manual mapping is labor-intensive.
• Automating health problem identification is crucial for improving efficiency and accuracy.

Purpose of the Study:

• To automate health problem identification from clinician-patient conversations using large language models (LLMs) with retrieval-augmented generation (RAG).
• To map identified health problems to the standardized Omaha System terminology.

Main Methods:

• Analysis of 5118 utterances from 22 home healthcare encounters using the Omaha System framework.
• Application of RAG-enhanced LLMs for detecting and mapping health problems.
• Evaluation of various LLM configurations (embedding models, context windows, parameters, prompting strategies) and three LLMs (Llama 3.1-8B-Instruct, GPT-4o-mini, GPT-o3-mini) using precision, recall, and F1-score.

Main Results:

• The optimal configuration involved a 1-utterance context window, top k=15, top p=0.6, and few-shot learning with chain-of-thought prompting.
• GPT-4o-mini achieved the highest F1-score (0.90) for both problem and sign/symptom identification.
• GPT-o3-mini and Llama 3.1-8B-Instruct showed lower performance (F1-scores of 0.83/0.82 and 0.73/0.72, respectively).

Conclusions:

• LLMs with RAG effectively automate health problem identification within the Omaha System framework in clinical conversations.
• This automation enhances documentation completeness and reduces the burden on clinicians.
• The approach has the potential to improve patient outcomes and clinical efficiency in home healthcare settings.