The Development of Complex Digital Health Solutions: Formative Evaluation Combining Different Methodologies

The development of digital health solutions for current health care settings requires an understanding of the complexities of the health care system, organizational setting, and stakeholder groups and of the underlying interplay between stakeholders and the technology. The digital health solution was founded on the basis of an information and communication technology platform and point-of-care devices enabling home-based monitoring of disease progression and treatment outcome for patients with rheumatoid arthritis (RA).

The aim of this paper is to describe and discuss the applicability of an iterative evaluation process in guiding the development of a digital health solution as a technical and organizational entity in three different health care systems.

The formative evaluation comprised the methodologies of contextual understanding, participatory design, and feasibility studies and included patients, healthcare professionals, and hardware and software developers. In total, the evaluation involved 45 patients and 25 health care professionals at 3 clinical sites in Europe.

The formative evaluation served as ongoing and relevant input to the development process of the digital health solution. Through initial field studies key stakeholder groups were identified and knowledge obtained about the different health care systems, the professional competencies involved in routine RA treatment, the clinics' working procedures, and the use of communication technologies. A theory-based stakeholder evaluation achieved a multifaceted picture of the ideas and assumptions held by stakeholder groups at the three clinical sites, which also represented the diversity of three different language zones and cultures. Experiences and suggestions from the patients and health care professionals were sought through participatory design processes and real-life testing and actively used for adjusting the visual, conceptual, and practical design of the solution. The learnings captured through these activities aided in forming the solution and in developing a common understanding of the overall vision and aim of this solution. During this process, the 3 participating sites learned from each other's feed-back with the ensuing multicultural inspiration. Moreover, these efforts also enabled the consortium to identify a 'tipping point' during a pilot study, revealing serious challenges and a need for further development of the solution. We achieved valuable learning during the evaluation activities, and the remaining challenges have been clarified more extensively than a single-site development would have discovered. The further obstacles have been defined as has the need to resolve these before designing and conducting a real-life clinical test to assess the outcome from a digital health solution for RA treatment.

A formative evaluation process with ongoing involvement of stakeholder groups from 3 different cultures and countries have helped to inform and influence the development of a novel digital health solution, and provided constructive input and feedback enabling the consortium to control the development process.