Development of an evidence-based evaluation framework for digital health software products

Current frameworks and standards for evaluation of digital health software products (DHSPs) are fragmented and may not cover the spectrum of what stakeholders consider most important. We conducted a mixed-methods study with the goal of producing an evidence-based and open-access evaluation framework (EF). We conducted a needs assessment among 173 subject matter experts (SMEs) to assess the need for this framework and identify its components using interviews, focus-group discussions, and a survey. Next, we conducted a narrative review of 1053 papers, and a landscape analysis of 160 relevant frameworks, guidances, and standards. This data was used to develop the components of the EF: high-level themes (domains), with evaluation criteria and associated benchmarks for evaluating the quality and trustworthiness of DHSPs. An additional 58 SME interviews were conducted to validate the EF. The top three domains for assessing overall quality were evidence, usability, and privacy and security. Equity was a common theme across all domains. The EF spans multiple domains of trust and value, harmonizes best practices, stands on the shoulders of well-respected and established work, and eases the effective adoption of high-quality, trustworthy DHSPs.