Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review

Area of Science:

• Artificial Intelligence in Medicine
• Healthcare Informatics
• Explainable AI (XAI)

Background:

• Agentic AI systems offer autonomous capabilities for healthcare but raise concerns about transparency and oversight.
• Existing research on explainable AI (XAI) primarily focuses on traditional predictive models, with limited understanding of its application in agentic architectures.

Purpose of the Study:

• To conduct a scoping review of explainable agentic AI (XAAI) in healthcare.
• To characterize the types, scope, and forms of explainability employed in these emerging systems.

Main Methods:

• Scoping review adhering to PRISMA-ScR guidelines.
• Searched major databases (PubMed, Embase, IEEE Xplore, ACM Digital Library) through November 2025.
• Extracted data on system architecture, explainability type, scope, form, and clinical outcomes for eligible studies.

Main Results:

• Nine studies included, all featuring agentic AI with autonomy and tool use, often in multi-agent systems.
• Explainability was mainly intrinsic and workflow-native, using textual reasoning traces and evidence grounding.
• XAAI systems showed improved performance and interpretability in radiology, neurology, psychiatry, and research, but studies were heterogeneous and lacked structured human oversight.

Conclusions:

• Current XAAI emphasizes process transparency and evidence grounding over mechanistic attribution.
• Limited and heterogeneous evidence suggests early trends, requiring cautious interpretation.
• Standardized evaluations, clear oversight reporting, and real-world validation are crucial for safe healthcare integration.