Retrieving Evidence from EHRs with LLMs: Possibilities and Challenges

Area of Science:

• Medical Informatics
• Artificial Intelligence in Medicine
• Radiology

Background:

• Unstructured data in Electronic Health Records (EHRs) contains vital diagnostic information often overlooked due to manual review limitations.
• Radiologists face time constraints, making it challenging to extract relevant evidence from extensive patient notes.

Purpose of the Study:

• To propose and evaluate a zero-shot strategy using large language models (LLMs) for efficient retrieval and summarization of unstructured EHR data.
• To assess the efficacy of LLMs in identifying and summarizing patient conditions and supporting evidence from clinical notes.

Main Methods:

• A zero-shot LLM strategy was developed to infer patient conditions and summarize supporting evidence from EHR notes.
• LLM-generated outputs were compared against a pre-LLM information retrieval baseline through expert evaluation.
• An LLM-based evaluation method was proposed and validated to scale up assessment of LLM outputs.

Main Results:

• LLM-based retrieval and summarization were consistently preferred by expert evaluators over the baseline method.
• LLM-generated outputs showed a strong correlation between model confidence and the faithfulness of the summarized evidence.
• The LLM-based evaluation method proved effective for scaling up the assessment process.

Conclusions:

• LLMs show significant promise as interfaces for accessing and interpreting unstructured data within EHRs for clinical decision support.
• While LLM
• hallucinations
• remain a challenge, model confidence can serve as a practical indicator to ensure summary accuracy.