Development and validation of the provider documentation summarization quality instrument for large language models.

Area of Science:

• Medical Informatics
• Artificial Intelligence in Healthcare
• Clinical Documentation

Background:

• Large language models (LLMs) are increasingly integrated into electronic health record (EHR) workflows.
• Validated instruments are crucial for evaluating LLM performance in healthcare settings.
• Existing documentation quality tools are often inadequate for LLM-generated text.

Purpose of the Study:

• To validate the Provider Documentation Summarization Quality Instrument-9 (PDSQI-9) for evaluating LLM-generated clinical summaries.
• To assess the construct validity of the PDSQI-9 across multiple dimensions.
• To provide a reliable instrument for assessing LLM performance in EHR documentation.

Main Methods:

• Generated multi-document summaries from real-world EHR data using various LLMs.
• Validated the PDSQI-9 using Pearson correlations, factor analysis, Cronbach's α, inter-rater reliability (ICC, Krippendorff's α), a semi-Delphi process, and discriminant validity tests.
• Physician raters underwent standardized training for consistent instrument application.

Main Results:

• The PDSQI-9 demonstrated strong internal consistency (Cronbach's α = 0.879) and high inter-rater reliability (ICC = 0.867).
• Factor analysis revealed a 4-factor model (organization, clarity, accuracy, utility) explaining 58% of the variance.
• The instrument showed good substantive validity, content relevance, and discriminant validity, distinguishing high- from low-quality summaries (P<.001).

Conclusions:

• The PDSQI-9 exhibits robust construct validity, high reliability, and a meaningful factor structure for evaluating LLM-generated clinical summaries.
• The instrument reliably captures key dimensions of documentation quality and distinguishes between high- and low-quality summaries.
• The PDSQI-9 is a practical tool for health systems to ensure safer and more effective integration of LLMs into clinical workflows.