Explainable Agentic Artificial Intelligence in Healthcare: A Scoping Review

Agentic artificial intelligence (AI) systems, characterized by autonomous goal-directed behavior, multi-step reasoning, task decomposition, and tool use, are increasingly proposed for healthcare applications. However, their autonomy raises concerns regarding transparency, accountability, and human oversight. While explainable AI (XAI) has been widely studied in traditional predictive models, less is known about how explainability is implemented within agentic architectures.

To map the emerging literature on explainable agentic AI (XAAI) in healthcare and characterize the types, scope, and forms of explainability used in these systems.

A scoping review was conducted following PRISMA-ScR guidelines. PubMed, Embase, IEEE Xplore, and ACM Digital Library were searched through November 2025. Eligible studies described healthcare-related agentic AI systems incorporating explicit explainability mechanisms. Data were extracted on system architecture, explainability type (intrinsic, post hoc, hybrid), explanation scope (local, global), explanation form, and reported clinical outcomes.

Nine studies met the inclusion criteria. All systems demonstrated core agentic features, including autonomy, task decomposition, and tool integration, often within multi-agent frameworks. Explainability was predominantly intrinsic and workflow-native, typically delivered through textual reasoning traces and example-based grounding in retrieved clinical evidence. Feature-based and global explanations were comparatively rare and largely confined to hybrid architectures. Across domains including radiology, neurology, psychiatry, and biomedical research, XAAI systems were reported to improve performance and interpretability relative to baseline models in the included studies. However, these findings were derived from heterogeneous, predominantly experimental or retrospective studies, and structured human-in-the-loop oversight was infrequently described.

Current XAAI systems appear to emphasize process transparency and evidence grounding rather than mechanistic model-level attribution. The available evidence remains limited and heterogeneous, and findings should be interpreted as early trends rather than established characteristics. Further progress will require standardized evaluation frameworks, clearer reporting of oversight mechanisms, and validation in real-world clinical settings to support safe and trustworthy integration of agentic AI into healthcare practice.