Testing and Evaluation of Generative Large Language Models in Electronic Health Record Applications: A Systematic...

Area of Science:

• Natural Language Processing
• Biomedical Informatics
• Artificial Intelligence in Healthcare

Background:

• Generative Large Language Models (LLMs) achieve high performance in NLP tasks.
• Clinical applications of LLMs with Electronic Health Records (EHRs) are limited and challenging.

Purpose of the Study:

• Systematically review generative LLM use with EHRs for patient care.
• Summarize challenges and suggest future research directions.

Main Methods:

• Searched PubMed and Web of Science for peer-reviewed articles since 2023.
• Included studies using generative LLMs to analyze real EHR data.
• Extracted data on prompt engineering, fine-tuning, multimodal data, and evaluation metrics.

Main Results:

• 76 studies analyzed; 88.2% used zero-shot prompting.
• Prompt engineering and fine-tuning showed performance improvements.
• Multimodal data use was limited (2 studies) but beneficial.
• Identified challenges: bias, hallucinations, impersonal tone, and lack of standardized metrics.

Conclusions:

• Few studies employ advanced techniques for LLM enhancement.
• Standardization of evaluation metrics is needed.
• Current LLMs cannot replace physicians due to bias, hallucinations, and impersonal responses.