Methodological issues in product evaluation: the influence of testing environment and task scenario

This article examines the utility of two commonly used approaches in the evaluation of interactive consumer products: lab-based testing and single task scenarios. These are compared to two more complex and resource-demanding approaches (field-based testing and dual task scenarios) with regard to the test results they produce. An experiment with N = 80 users was carried out, employing a 2 (laboratory vs. field) by 2 (single task vs. dual task scenario) by 2 (on-product information: present vs. absent) between-subjects design. On-product information (advising users to save water and electricity during kettle usage) represented the intervention, of which the effects on user behaviour were compared under the different experimental conditions. The main finding was that the impact of on-product information on user behaviour was strongest in the lab-based testing environment using a single task scenario (i.e., most economical testing condition), compared to the three other experimental conditions. The work found similar effects for self-report measures. The findings of the study point to the risk that the effects of system redesign on user behaviour may be overestimated if low-fidelity testing approaches are employed. The relevance of these findings for other application areas is also discussed (e.g., design of warnings).