Methodological issues in product evaluation: the influence of testing environment and task scenario

Area of Science:

• Human-Computer Interaction
• Usability Engineering
• Consumer Product Design

Background:

• Evaluating interactive consumer products often involves trade-offs between testing fidelity and resource demands.
• Commonly used methods include lab-based testing and single-task scenarios, which are resource-efficient.
• More complex methods like field-based testing and dual-task scenarios offer higher fidelity but require more resources.

Purpose of the Study:

• To compare the utility of low-fidelity (lab, single-task) versus high-fidelity (field, dual-task) testing approaches.
• To evaluate how different testing conditions affect the observed impact of on-product information on user behavior.
• To assess the reliability of self-report measures across various testing scenarios.

Main Methods:

• A between-subjects experiment with 80 users was conducted.
• A 2x2x2 factorial design was employed: laboratory vs. field, single task vs. dual task, and on-product information present vs. absent.
• The intervention involved on-product information designed to encourage water and electricity conservation during kettle use.

Main Results:

• The impact of on-product information on user behavior was significantly stronger in the lab-based, single-task scenario compared to other conditions.
• Similar effects were observed for self-report measures, indicating potential biases in user perception.
• Low-fidelity testing approaches may lead to an overestimation of system redesign effects on user behavior.

Conclusions:

• Lab-based, single-task testing may overestimate the effectiveness of interventions like on-product information.
• Findings suggest caution when generalizing results from low-fidelity testing to real-world user behavior.
• The study highlights the importance of choosing appropriate testing methodologies to avoid overestimating design impacts, with implications for areas like warning design.