Improving electronic health record processing of large language models via retrieval-augmented generation: A case...

Area of Science:

• Natural Language Processing (NLP)
• Clinical Informatics
• Artificial Intelligence (AI)

Background:

• Large language models (LLMs) demonstrate significant capabilities in natural language processing (NLP).
• However, LLMs face challenges in extracting domain-specific information from complex electronic health records (EHRs).
• Dietary supplement (DS) data within EHRs presents unique extraction complexities.

Purpose of the Study:

• To evaluate the effectiveness of retrieval-augmented generation (RAG) in enhancing LLMs for dietary supplement (DS) information extraction from EHRs.
• To compare the impact of different retrieval strategies against model size and domain specialization.

Main Methods:

• LLMs, including Llama-3, were tested with various retrievers on NLP tasks such as entity recognition and usage classification.
• Task-aligned retrieval methods were compared with general retrieval and model-centric approaches.
• Performance was assessed based on the accuracy of information extraction and classification.

Main Results:

• Task-aligned retrieval strategies significantly outperformed reliance on LLM size or domain specialization.
• Smaller, general-purpose LLMs combined with optimized retrievers achieved performance comparable to or exceeding specialized models.
• Structured retrieval improved performance on complex tasks like triple extraction, while semantic retrieval enhanced classification accuracy.

Conclusions:

• Retrieval strategies are more critical than model size or specialization for effective EHR analysis using LLMs.
• RAG offers a practical and cost-effective approach for clinical NLP, enabling efficient EHR data extraction without requiring massive computational resources.
• This research advances tools for evidence-based healthcare by emphasizing adaptable knowledge integration for real-world medical applications.