Explainable artificial intelligence in clinical decision support systems: a systematic review of techniques, clinical...

Explainable artificial intelligence (XAI) has emerged as an important approach for improving the transparency and interpretability of artificial intelligence (AI)-based clinical decision support systems (CDSSs). However, evidence regarding the models and explainability techniques used in healthcare, their clinical evaluation, methodological quality, and progression toward real-world implementation remains fragmented. This systematic review aimed to synthesize and critically evaluate the current evidence on XAI-enabled CDSSs, with particular emphasis on AI and XAI methods, clinical applications, validation, explainability evaluation, implementation, and methodological limitations.

A systematic review was conducted in accordance with the PRISMA 2020 statement. The protocol was prospectively registered in PROSPERO (CRD420261449460). PubMed/MEDLINE, Scopus, and Web of Science Core Collection were searched from database inception through 30 June 2026 (searches run on 15 July 2026). Eligible studies were peer-reviewed original research articles that applied artificial intelligence within healthcare clinical decision support systems, incorporated at least one explainability or interpretability technique, and demonstrated a direct role in clinical decision-making or decision support. Studies limited to prediction, classification, diagnosis, or prognostic modeling without a decision-support component were excluded. Two reviewers jointly screened titles and abstracts, independently assessed full texts, and independently extracted data using a standardized form informed by the CHARMS checklist. Risk of bias and applicability of prediction model studies were assessed using PROBAST, and reporting completeness was evaluated against relevant TRIPOD + AI recommendations where applicable. Due to substantial clinical, methodological, and statistical heterogeneity, findings were synthesized narratively in accordance with SWiM guidance.

Of 2008 identified records, 68 studies met the eligibility criteria and were included in the qualitative synthesis. Publications increased substantially in 2025-2026, accounting for 47 of the 68 included studies. China and India contributed the largest number of studies (11 each), followed by the United States (10). Critical care/intensive care medicine and cardiology were the most frequently represented clinical specialties (13 studies each). Numeric patient-level sample size was reported in 35 studies. Among these, the median sample size was 973 (IQR 303-10,015; range 57-106,449), while the mean was 11,944, indicating a markedly right-skewed distribution. Internal validation was reported in 51 studies (75.0 %), whereas external validation was reported in 17 studies (25.0 %). Most systems remained at the research-prototype stage (n = 63, 92.6 %). One study (1.5 %) reported pilot testing in a clinical workflow, while only two studies (2.9 %) reported real-world or routine clinical deployment. Implementation status was unclear or not reported in two studies (2.9 %). Of the 63 studies for which PROBAST was applicable, 3 (4.8 %) were judged at low overall risk of bias and 3 (4.8 %) at high risk; the remainder were rated Unclear mainly because of incomplete reporting. Five studies describing non-prediction CDSS designs were classified as PROBAST not applicable. Clinician involvement was reported in 53 studies. Formal user studies with structured evaluation protocols were reported in 39 studies (57.4 %), but rigorous assessment of effects on decision quality or patient outcomes remained rare.

XAI-enabled CDSS research has expanded rapidly, yet remains largely confined to model development and technical explainability rather than rigorous clinical translation. Although SHAP and gradient-boosting methods currently predominate, external validation, real-world deployment, standardized evaluation of explanation quality, and meaningful human-centered assessment are still limited. Future work must prioritize multicenter external validation, transparent reporting, rigorous assessment of explanation fidelity and stability, active clinician and patient involvement, and prospective evaluation of effects on decision-making and patient outcomes. Closing the gap between algorithmic performance and clinically meaningful utility is essential for the safe, trustworthy, and responsible adoption of these systems.