Validation of system usability scale as a usability metric to evaluate voice user interfaces

Area of Science:

• Human-Computer Interaction
• Interactive Systems Design
• Usability Engineering

Background:

• User experience (UX) is crucial for interactive systems, yet evaluating voice user interfaces (VUIs) lags behind graphical user interfaces (GUIs). Standardized tools for VUI evaluation are underdeveloped.
• The System Usability Scale (SUS) is a well-established metric for assessing the usability of interactive systems, primarily for GUIs.
• Voice assistants like Amazon Alexa and Google Nest Mini are increasingly prevalent, necessitating robust evaluation methods.

Purpose of the Study:

• To evaluate the usability of two prominent voice assistants, Amazon Alexa Echo Dot and Google Nest Mini, using the System Usability Scale (SUS).
• To compare the SUS scores and confidence intervals of these voice assistants.
• To analyze the Adjective Rating Score (ARS) for qualitative insights and validate its correlation with SUS scores in the context of VUIs.

Main Methods:

• A usability evaluation was conducted with 16 participants performing predefined tasks on Amazon Alexa Echo Dot and Google Nest Mini.
• The System Usability Scale (SUS) questionnaire was administered post-task completion.
• Adjective Rating Scores (ARS) were analyzed, and a paired sample t-test was used to compare the SUS scores between the two devices.

Main Results:

• Significant differences were observed in the SUS scores between the Amazon Alexa Echo Dot and Google Nest Mini.
• Confidence intervals were derived for both devices, providing a range for their true usability scores.
• A correlation was found between the SUS scores and the Adjective Rating Scores, offering complementary qualitative data.

Conclusions:

• The System Usability Scale (SUS) is a viable and effective metric for evaluating the usability of voice user interfaces (VUIs).
• The study highlights significant usability differences between leading voice assistants.
• Researchers are encouraged to adopt SUS for consistent and reliable VUI usability assessment.