Adapting Generative Large Language Models for Information Extraction from Unstructured Electronic Health Records in...

Area of Science:

• Artificial Intelligence
• Health Informatics
• Natural Language Processing

Background:

• Extracting information from unstructured electronic health records is complex.
• Generative large language models (LLMs) offer potential solutions for clinical information extraction.
• Optimal LLM adaptation methods for residential aged care settings require further investigation.

Purpose of the Study:

• To evaluate zero-shot and few-shot learning methods for LLMs in aged care.
• To assess the impact of parameter-efficient fine-tuning (PEFT) and retrieval-augmented generation (RAG) on LLM performance.
• To compare the effectiveness of different LLM training strategies for named entity recognition (NER) in nursing notes.

Main Methods:

• Utilized Llama 3.1-8B for named entity recognition (NER) on nursing notes from Australian aged care facilities.
• Compared zero-shot and few-shot learning with and without PEFT and RAG.
• Evaluated performance using accuracy, precision, recall, and F1 scores, with statistical significance testing.

Main Results:

• Zero-shot and few-shot learning with PEFT or RAG showed comparable performance.
• Few-shot learning outperformed zero-shot learning without PEFT or RAG.
• PEFT significantly improved performance for both zero-shot and few-shot learning.
• RAG significantly improved performance only for few-shot learning.
• Zero-shot learning with PEFT achieved performance comparable to few-shot learning with RAG.

Conclusions:

• PEFT is a highly effective method for adapting LLMs for clinical information extraction in aged care.
• Few-shot learning combined with RAG offers superior performance.
• Findings guide the optimization of LLMs for clinical IE in aged care settings.