Standards as applied in reality: a case study on the translation of standards in eHealth evaluation practice

Application of standards is a way to increase quality in an evaluation study. However, standards are used insufficiently in eHealth evaluation, affecting the generalization of the knowledge generated. This study aimed to explore how standards are used in a practical setting of an eHealth evaluation, and to identify the factors that can hinder their use.

The data were collected in a multi-national and interdisciplinary eHealth evaluation study targeted at the elderly people suffering from mild cognitive impairment and mild dementia. The study was carried out in four countries and funded by the European Union. The collected data included meeting minutes (n = 8) and e-mail correspondence (n = 261) between partners. The chronological sequence of events related to the use of standards was established. Subsequently, the hindering factors related to the use of standards were identified from the sequence.

The use of four standards was described, reflecting a variety of related processes or barriers that emerge during eHealth evaluation. The processes differed based on the type of the standard. Specifically, evaluation frameworks were found to be conceptual standards and they were easy to agree upon, while standardized metrics were more tangible and their use triggered negotiations. Further, the following factors hindered the use of standards in eHealth evaluations: (1) inadequacy of a standard to address a target population or a disease, (2) insufficient resources to use a standard, (3) lack of experience in using a standard, and (4) lack of validation of a standard in a particular location.

Standardization initiatives in eHealth evaluation provide a blueprint for evaluation, but their practical application is problematic. The practical circumstances of an evaluation study can cause deviations in the standards, thus producing heterogeneity in the evaluation methodologies.