Verifying Facts in Patient Care Documents Generated by Large Language Models Using Electronic Health Records

Large language models (LLMs) can generate clinical documents, but inaccuracies due to hallucinations create a barrier for use in medical practice. Reliance on clinician vigilance to remedy errors is infeasible, since mistakes may be subtle and interleaved among accurate statements within the document. An increasing trend in artificial intelligence (AI) has been to deploy LLMs as evaluators of LLM output (LLM-as-a-Judge), which has been shown to have high concordance with human judgment.

We hypothesize that an LLM-as-a-Judge with relevant facts from a patient's electronic health record (EHR) can be used to automatically verify facts in any clinical document written about a patient and provide an overall estimate of the document's accuracy. To evaluate this hypothesis, we created a benchmark dataset called VeriFact - Brief Hospital Course (VeriFact-BHC), consisting of statements derived from the brief hospital course sections of discharge summaries, and established a ground truth from clinician chart review. We then devised an AI system, VeriFact, that combines retrieval-augmented generation of facts from a patient's EHR along with an LLM-as-a-Judge to perform automated evaluation of all statements in the dataset.

VeriFact-BHC contains 100 patients with 13,070 statements derived from brief hospital courses, each annotated by three or more clinicians. VeriFact achieved 93.2% agreement with clinicians. The highest interrater agreement among clinicians was 88.5%, indicating that VeriFact can produce more consistent fact verification than humans. Scaling the size of the LLM-as-a-Judge model and adding reasoning capability improve different facets of AI judgment and are synergistic when combined.

VeriFact can help clinicians verify facts in documents drafted by LLMs prior to committing them to the patient's EHR and can automate tasks requiring chart review. VeriFact-BHC can be used to develop and benchmark new methodologies for verifying facts in patient care documents. (Funded by Stanford University and others.).