Approaches to Evaluating Digital Health Technologies: Scoping Review

Profound scientific evaluation of novel digital health technologies (DHTs) is key to enhance successful development and implementation. As such, we previously developed the eHealth evaluation cycle. The eHealth evaluation cycle contains 5 consecutive study phases: conceptual, development, feasibility, effectiveness, and implementation.

The aim of this study is to develop a better understanding of the daily practice of the eHealth evaluation cycle. Therefore, the objectives are to conduct a structured analysis of literature data to analyze the practice of the evaluation study phases and to determine which evaluation approaches are used in which study phase of the eHealth evaluation cycle.

We conducted a systematic literature search in PubMed including the MeSH term "telemedicine" in combination with a wide variety of evaluation approaches. Original peer-reviewed studies published in the year 2019 (pre-COVID-19 cohort) were included. Nonpatient-focused studies were excluded. Data on the following variables were extracted and systematically analyzed: journal, country, publication date, medical specialty, primary user, functionality, evaluation study phases, and evaluation approach. RStudio software was used to summarize the descriptive data and to perform statistical analyses.

We included 824 studies after 1583 titles and abstracts were screened. The majority of the evaluation studies focused on the effectiveness (impact; 304/824, 36.9%) study phase, whereas uptake (implementation; 70/824, 8.5%) received the least focus. Randomized controlled trials (RCTs; 170/899, 18.9%) were the most commonly used DHT evaluation method. Within the effectiveness (impact) study phase, RCTs were used in one-half of the studies. In the conceptual and planning phases, survey research (27/78, 35%) and interview studies (27/78, 35%) were most frequently used. The United States published the largest amount of DHT evaluation studies (304/824, 36.9%). Psychiatry and mental health (89/840, 10.6%) and cardiology (75/840, 8.9%) had the majority of studies published within the field.

We composed the first comprehensive overview of the actual practice of implementing consecutive DHT evaluation study phases. We found that the study phases of the eHealth evaluation cycle are unequally studied and most attention is paid to the effectiveness study phase. In addition, the majority of the studies used an RCT design. However, in order to successfully develop and implement novel DHTs, stimulating equal evaluation of the sequential study phases of DHTs and selecting the right evaluation approach that fits the iterative nature of technology might be of the utmost importance.