User-Centered Methods in Explainable AI Development for Hospital Clinical Decision Support: A Scoping Review

Area of Science:

• Health Informatics
• Artificial Intelligence in Medicine
• Human-Computer Interaction

Background:

• Explainable Artificial Intelligence (XAI) enhances transparency and usability of AI-based Clinical Decision Support Systems (CDSS).
• Effective integration of XAI into clinical workflows necessitates comprehensible, relevant, and actionable explanations for healthcare professionals.
• XAI-CDSS tools aim to improve diagnosis, treatment planning, and risk prediction in healthcare.

Purpose of the Study:

• To investigate the extent and methods of end-user involvement in the design and development of XAI-based CDSS for hospital settings.
• To identify common practices and gaps in engaging healthcare professionals during the development of XAI-CDSS.

Main Methods:

• A systematic scoping review of literature from Medline, Embase, and Web of Science.
• Inclusion criteria focused on studies detailing end-user involvement in XAI-CDSS design.
• Analysis of 11 identified studies, including qualitative methods like interviews and focus groups.

Main Results:

• Interviews and focus groups, primarily with physicians, were common engagement methods.
• Only four studies involved users across multiple development stages and tested various explanation techniques.
• Quality assessment revealed limitations in recruitment strategies and analysis detail in some studies.

Conclusions:

• Early and continuous engagement of diverse end-users, including physicians and nurses, is vital for effective XAI-CDSS development.
• Testing different explanation techniques with end-users is necessary to ensure cognitive alignment and usability.
• Future research should prioritize robust user involvement strategies and detailed reporting of engagement methods.