AI-driven analysis of patient safety reports using large language models: an exploratory multiple methods study

Area of Science:

• Artificial Intelligence in Healthcare
• Natural Language Processing
• Patient Safety Informatics

Background:

• Healthcare organizations struggle to analyze high volumes of patient safety event reports.
• Manual review of narrative data is time-consuming and resource-intensive, leaving most low-harm events unexamined.
• Large language models (LLMs) present a novel technological solution to this data analysis challenge.

Purpose of the Study:

• To develop and evaluate an AI-driven approach using LLMs for patient safety report analysis.
• To identify patient safety issues and uncover system-level trends within a US healthcare system.
• To assess the readiness for implementing LLM technology in healthcare settings.

Main Methods:

• Quantitative evaluation of OpenAI's GPT-4o model for accuracy in analyzing 9357 patient safety reports.
• LLM extracted safety problems, generated a taxonomy, and labeled reports; patient safety experts validated accuracy.
• Qualitative assessment of dashboard acceptability, appropriateness, and adoption through 10 stakeholder interviews.

Main Results:

• The LLM achieved high agreement scores: 94% for problem identification, 91.5% for parent categories, and 83.3% for child categories.
• Previously hidden patterns of patient safety issues were identified by the LLM.
• Stakeholders found LLM-generated insights valuable, appropriate, and perceived few barriers to adoption, anticipating expedited reviews and improved quality improvement.

Conclusions:

• LLMs can effectively capture and analyze complex concepts within patient safety reports.
• By summarizing and categorizing problems, LLMs reveal unidentified trends and system-level risks.
• LLMs can augment manual reviews, expedite safety event analysis, and guide quality improvement interventions.