Electronic Health Record User-driven Evaluations and Adaptations of Large Language Models for Clinical Applications

Large language models (LLMs) are emerging as generalist tools capable of interpreting clinical language and supporting healthcare professionals across diverse medical contexts and tasks, making electronic health records (EHRs) a natural and powerful setting for their deployment. Yet realizing this potential safely and effectively in real-world practice requires solutions adapted to EHR users' needs and context.

To propose a new paradigm for medical LLM development and evaluation that places EHR users, data, and EHR infrastructure at its center.

In this narrative review, we propose three mutually reinforcing pillars of the EHR ecosystem: the daily users of these systems, such as physicians and nurses; the rich and detailed EHR data produced through routine clinical care; and the EHR system as institutional and technical infrastructure. We then examine the role these pillars play in LLM development and evaluation.

We show that daily EHR users provide essential input for aligning LLMs with real clinical needs; EHR data offer the most meaningful foundation for model alignment and assessment; and the EHR system as infrastructure shapes what effective and responsible deployment looks like. Together, these pillars are shown to play an essential role in developing trustworthy and useful LLMs adapted to clinical applications, conducting reliable and meaningful evaluations, and assessing the impact of LLMs in real healthcare settings.

A paradigm centered on EHR users, data, and systems is needed to develop, evaluate, and deploy LLMs that are trustworthy, useful, and adapted to real-world clinical practice.