Automating Evaluation of AI Text Generation in Healthcare with a Large Language Model (LLM)-as-a-Judge

Area of Science:

• Artificial Intelligence in Healthcare
• Clinical Informatics
• Natural Language Processing

Background:

• Electronic Health Records (EHRs) contain complex data, challenging for providers to synthesize.
• Generative AI and Large Language Models (LLMs) offer automated summarization to reduce provider cognitive load.
• Accurate evaluation of LLM-generated summaries is essential for clinical safety and reliability.

Purpose of the Study:

• To introduce and validate an automated method for evaluating EHR multi-document summaries using an LLM as a judge (LLM-as-a-Judge).
• To assess the reliability and efficiency of LLM-as-a-Judge compared to human expert evaluations.

Main Methods:

• Developed and validated an LLM-as-a-Judge framework for evaluating EHR summaries.
• Benchmarked the framework against the Provider Documentation Summarization Quality Instrument (PDSQI)-9.
• Evaluated performance using metrics like intraclass correlation coefficient (ICC) and score differences.

Main Results:

• The LLM-as-a-Judge framework demonstrated strong inter-rater reliability with human evaluators (GPT-o3-mini ICC: 0.818).
• LLM-as-a-Judge achieved a median score difference of 0 and evaluation times of 22 seconds.
• Reasoning LLMs excelled in evaluations requiring advanced reasoning and domain expertise, outperforming other models.

Conclusions:

• Automated LLM-as-a-Judge provides a scalable and efficient solution for evaluating AI-generated medical summaries.
• This method ensures the accuracy and safety of LLM-generated clinical documentation summaries.
• LLM-as-a-Judge facilitates rapid identification of reliable AI summaries in healthcare settings.