MedFactEval and MedAgentBrief: A Framework and Workflow for Generating and Evaluating Factual Clinical Summaries

Area of Science:

• Artificial Intelligence
• Medical Informatics
• Clinical Natural Language Processing

Background:

• Evaluating factual accuracy in clinical text generated by Large Language Models (LLMs) is crucial for adoption.
• Manual expert review is not scalable for continuous quality assurance of LLM-generated clinical content.

Purpose of the Study:

• To introduce MedFactEval, a scalable framework for fact-grounded evaluation of LLM-generated clinical text.
• To present MedAgentBrief, a workflow for generating high-quality, factual clinical discharge summaries.
• To validate the MedFactEval framework against a gold standard established by human expert consensus.

Main Methods:

• MedFactEval utilizes clinician-defined key facts and an "LLM Jury" (multi-LLM majority vote) for evaluation.
• MedAgentBrief is a model-agnostic, multi-step workflow for generating discharge summaries.
• A gold standard was created using a seven-physician majority vote on key facts from inpatient cases.

Main Results:

• The MedFactEval LLM Jury demonstrated near-perfect agreement with the human expert panel (Cohen's κ = 81%).
• The LLM Jury's performance was statistically non-inferior to that of a single human expert (κ = 67%, P < 0.001).

Conclusions:

• MedFactEval offers a robust and scalable framework for evaluating factual accuracy in clinical LLM outputs.
• MedAgentBrief provides a high-performing generation workflow for factual clinical summaries.
• The combined approach facilitates the responsible deployment of generative AI in clinical settings.