Large Language Models for More Efficient Reporting of Hospital Quality Measures

Area of Science:

• Health Informatics
• Artificial Intelligence in Healthcare
• Quality Improvement

Background:

• Hospital quality measures are crucial for learning health systems but face challenges with cost, statistical power, and interrater reliability.
• Large language models (LLMs) show potential for automating healthcare tasks, including large-scale data abstraction from clinical records.

Purpose of the Study:

• To evaluate the performance of an LLM-based system for abstracting the Severe Sepsis and Septic Shock Management Bundle (SEP-1) quality measure.
• To assess the accuracy and reliability of LLM-driven data abstraction compared to manual methods using electronic health record data.

Main Methods:

• An LLM system was developed to ingest Fast Healthcare Interoperability Resources (FHIR) data and output SEP-1 abstractions.
• The LLM system was tested on 100 manual SEP-1 abstractions from University of California San Diego Health.
• Agreement between the LLM system and manual abstractors was calculated, with discordant cases reviewed by experts.

Main Results:

• The LLM system achieved 90% agreement (κ=0.82) with manual abstractors on SEP-1 measure category assignment.
• Expert review of discordant cases revealed that four were errors in the original manual abstraction.
• This indicates a high level of accuracy for the LLM system in abstracting complex quality measures.

Conclusions:

• LLM-based systems show potential for accurate and reliable abstraction of complex hospital quality measures using interoperable EHR data.
• This technology may offer a scalable solution to improve the efficiency and consistency of quality measure reporting in healthcare.