In Silico Evaluation of Algorithm-Based Clinical Decision Support Systems Based on Care Pathway Simulation Models:...

In silico evaluation (ISE) methods create a digital twin or a computer simulation of actual care pathways, enabling a broader assessment of the potential impact of algorithm-based clinical decision support systems (CDSS) before implementation. A programmatic search of several academic research databases showed at least 886 CDSS development and evaluation studies in the past 3 decades. However, fewer than 3% applied ISE to evaluate the potential impact on broader clinical care pathways.

This study aims to review the scope of proposed ISE methods to evaluate CDSS, with a focus on simulation modeling approaches, care pathway parameters considered, and outcomes evaluated within the ISE methodological domain.

This review followed the established scoping review methodological guidelines. We conceptualized a tailored search framework and conducted a 2-stage screening process on studies identified through automated searches of selected databases. Relevant information on CDSS study characteristics and the application of care pathway simulation modeling in CDSS evaluation was subsequently extracted.

A small subset of studies on CDSS development conducted ISE. Most ISE studies were published after 2019, reflecting a more recent increase in the application of ISE. These studies frequently emphasized patient, process, and cost-effectiveness outcomes. Notably, the evaluation of outcomes directly related to care providers' well-being is lacking, highlighting a critical gap in current ISE applications. Among the studies included in this review, various simulation modeling paradigms were used, including dynamic simulations and state-based models. Three themes were found among the different motivations and objectives for using ISE: (1) outcome comparison, (2) outcome comparison with sensitivity analysis, and (3) simulation-based optimization of the proposed CDSS. The first two approaches considered a decoupled CDSS model training followed by simulation-based evaluation of the trained CDSS model. The third approach iteratively improved the CDSS model's decision-support capabilities through optimization based on care pathway simulations. These approaches can be broadly categorized as decoupled (1 and 2), where CDSS models are evaluated within seperately designed clinical workflow simulations, and integrated (3), where simulation iteratively optimizes CDSS performance.

The growing body of algorithm-based CDSS research underscores the need for evaluation approaches that are resource-efficient and account for systems-level workflow implications. This review highlights both the gaps and the potential of ISE, particularly care pathway simulation-based approaches, as a preimplementation strategy to strengthen evidence before significant resource allocations to pilot or full-scale implementations. ISE presents as a promising intermediary evaluation approach that bridges model-level performance and clinical workflow impact, allowing more contextualized and resource-efficient assessments prior to implementation.