Safety design guidelines for clinician-AI interaction in computer-aided diagnosis systems using system-theoretic...

Area of Science:

• Medical Informatics
• Human-Computer Interaction
• Artificial Intelligence Safety

Background:

• Patient safety is paramount in AI-enabled Computer-Aided Diagnosis (CADx).
• Current safety frameworks inadequately address human-AI interaction complexities in clinical settings.
• Underexplored human-centered safety challenges in AI diagnostic tools pose risks.

Purpose of the Study:

• To develop a systematic, human-centered approach for safety guidelines in clinician-AI interaction for CADx systems.
• To identify and mitigate hazards associated with AI collaboration in diagnostic decision-making.
• To enhance the reliability and safety of AI-driven diagnostic tools.

Main Methods:

• System-Theoretic Process Analysis (STPA) was employed to identify critical hazards in clinician-AI collaboration.
• Safety design guidelines were formulated based on identified unsafe control actions.
• A CADx interaction framework with a safety-oriented GUI and multiple explanation methods was developed.
• A safety-oriented evaluation approach using explanation consistency was introduced.

Main Results:

• Critical hazards like automation bias and misinterpretation of AI explanations were identified.
• Actionable safety design guidelines promoting transparency, coherent explanations, and calibrated trust were formulated.
• A novel interaction framework operationalized guidelines, enhancing clinician engagement.
• An evaluation method using explanation consistency was proposed to detect unreliable AI explanations.

Conclusions:

• The proposed unified framework effectively links safety analysis, interaction design, and explainability evaluation for AI in CADx.
• This approach provides a reusable method for improving the safety and reliability of AI-driven diagnostic systems.
• Prioritizing human-centered design and robust evaluation is crucial for safe AI implementation in healthcare.