Development of a Preliminary Patient Safety Classification System for Generative AI

Area of Science:

• Healthcare Technology
• Artificial Intelligence
• Patient Safety

Background:

• Generative artificial intelligence (AI) offers transformative potential in healthcare.
• Effective classification and monitoring of AI-related patient safety risks are crucial.
• Existing frameworks for AI safety in healthcare are limited.

Purpose of the Study:

• To develop and evaluate a preliminary classification system for generative AI patient safety errors.
• To assess the system's utility in categorizing errors from diverse AI healthcare applications.
• To establish a foundation for robust AI error monitoring in clinical settings.

Main Methods:

• A novel classification system was designed, structured around AI input and output stages with specific error typologies.
• The system was applied to two distinct generative AI applications: patient-facing large language models (LLMs) and an ambient digital scribe (ADS) system.
• Error categorization focused on type, frequency, and clinical significance.

Main Results:

• In LLM analysis (27 queries), 45 errors were identified, with omissions being most frequent (42%); 25% of errors were high clinical significance.
• In ADS simulation (11 visits), 66 errors were identified, with omissions predominating (83%); no high clinical significance errors were noted.
• The classification system effectively categorized output errors across both LLM and ADS applications.

Conclusions:

• The developed classification system demonstrates utility in identifying and categorizing generative AI patient safety errors in healthcare.
• Omission errors were prevalent in both analyzed AI applications, highlighting a key area for improvement.
• This system provides a foundational framework for enhancing the safety and reliability of AI in clinical practice.