From expert-derived user needs to user-perceived ease of use and usefulness: a two-phase mixed-methods evaluation...

Area of Science:

• Human-Computer Interaction
• Health Informatics
• Software Engineering

Background:

• Technology evaluation faces barriers like underspecified user needs and lack of a gold standard.
• Existing methods struggle to assess novel technologies when user needs are difficult to articulate.

Purpose of the Study:

• To present a two-phase evaluation framework addressing technology assessment challenges.
• To integrate usability experts and end-users for comprehensive evaluation.
• To enable effective assessment of user needs satisfaction and perceived usability of novel designs.

Main Methods:

• A two-phase framework: Phase 1 (usability experts) and Phase 2 (end-users).
• Phase 1 involves functionality alignment and cognitive walkthroughs.
• Phase 2 employs mixed methods: time-motion analysis, software logs, usability questionnaires (System Usability Scale, Unified Theory of Acceptance of Use of Technology), think-aloud protocols, and interviews.

Main Results:

• Triangulation of quantitative and qualitative data provides insights into ease-of-use, functionality integration, user anxiety, and workflow impact.
• The framework was successfully applied in a formative evaluation of the IMPACT software.
• Demonstrates effective assessment of user needs satisfaction and perceived usefulness and usability.

Conclusions:

• The mixed-methods evaluation framework facilitates an integrated assessment of novel technology.
• It bridges the gap between evolving user needs and technology design in iterative prototyping.
• Particularly valuable for technologies where users struggle to articulate needs due to lack of a baseline.