Are UX Evaluation Methods Providing the Same Big Picture?

Area of Science:

• Human-Computer Interaction (HCI)
• Software Engineering
• User Experience (UX) Research

Background:

• Software application success hinges on user retention, making User eXperience (UX) evaluation critical.
• Existing UX evaluation methods yield varied or contradictory results, complicating practitioner choices.
• Previous studies often overlook developer perspectives and the longitudinal nature of UX.

Purpose of the Study:

• To conduct a longitudinal study comparing different UX evaluation methods.
• To investigate the impact of contrasting UX evaluation results on developer decision-making.
• To highlight the variability of UX evaluation methods and their influence on software development.

Main Methods:

• A longitudinal study involving 68 students evaluating an online judge system.
• Utilized AttrakDiff, User Experience Questionnaire (UEQ), and Sentence Completion methods.
• Data collected at three distinct time points over a semester.

Main Results:

• Contrasting results were observed between the AttrakDiff, UEQ, and Sentence Completion methods.
• These discrepancies influenced developers' interpretations and decision-making processes.
• The study demonstrated that different methods provide divergent views of user experience over time.

Conclusions:

• The choice of UX evaluation method significantly impacts study outcomes and subsequent development decisions.
• A longitudinal approach is crucial for understanding the dynamic nature of user experience.
• The HCI community must address the discrepancies between UX evaluation methods and their practical implications.