An Explanation User Interface for Artificial Intelligence-Supported Mechanical Ventilation Optimization for...

The integration of artificial intelligence (AI) into clinical decision support systems (CDSSs) for mechanical ventilation in intensive care units (ICUs) holds great potential. However, the lack of transparency and explainability hinders the adoption of opaque AI models in clinical practice. Explanation user interfaces (XUIs), incorporating explainable AI algorithms, are considered a key solution to enhance trust and usability. Despite growing research on explainable AI in health care, little is known about how clinicians perceive and interact with such explanation interfaces in high-stakes environments such as the ICU. Addressing this gap is essential to ensure that AI-supported CDSS are not only accurate but also trusted, interpretable, and seamlessly integrated into clinical workflows.

This study aimed to evaluate the first iteration of the design and evaluation phase of an XUI for an AI-based CDSS intended to optimize mechanical ventilation in the ICU. Specifically, it explores how different user groups-ICU nurses and physicians-perceive and prioritize explanation concepts, providing the empirical foundation for subsequent refinement iterations.

A midfidelity prototype was developed using the prototyping software Justinmind, based on existing guidelines, scientific literature, and insights from previous user-centered design (UCD) phases. The design process followed ISO (International Organization for Standardization) 9241-210 principles for UCD and combined qualitative and quantitative feedback to identify usability strengths, design challenges, and role-specific explanation needs. The prototype was evaluated formatively through 2 usability walkthroughs (walkthrough 1: 4 resident physicians and walkthrough 2: 4 ICU nurses), which included guided group discussions and Likert-scale assessments of explanation concepts in terms of understandability, suitability, and visual appeal.

The XUI was structured into 2 levels: a first level displaying high-level explanations (outlier warning and output certainty) alongside the CDSS output, and a second level offering more detailed explanations (available input, feature importance, and rule-based explanation) for users seeking deeper insight. While both user groups appreciated the first level, physicians found the second level of the XUI useful, whereas ICU nurses found it overly detailed. Thus, the structure was able to address the differing needs for explanations. The layered design helped balance transparency and information overload by providing initially concise explanations and more detailed ones on demand. The evaluation further strengthened evidence for role-dependent explanation needs, suggesting that nurses prefer actionable, concise insights, whereas physicians benefit from more granular transparency information.

This study underscores the importance of UCD in designing XUIs for CDSS. It highlights the differing information needs of physicians and ICU nurses, emphasizing the value of involving users early in the development of suitable XUIs. The findings provide practical guidance for designing layered, role-sensitive explanation interfaces in critical care and form the basis for future iterative evaluations and experimental studies assessing their impact on decision-making and clinician trust.